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ABSTRACT 

The monograph advances a conceptual framework for the 
evaluation of mental health services provided to chronically mentally 
ill persons by psychosocial rehabilitation programs. Evaluation 
mechanisms consistent with the philosophy and programmatic aims of 
psychosocial rehabilitation programs are identified. An introductory 
section provides a brief history of the psychosocial rehabilitation 
movement, proposes a con jeptual framework, and highlights existing 
evaluation publications which nay be of use to program 
administrators. The next five sections describe components of a 
comprehensive evaluation strategy, focusing on the following 
dimensions of psychosocial rehabilitation programs: (1) input (agency 
mission, client characteristics and ne^ds, staff competencies, 
financial condition); (2) process (program practices, program 
connectedness, match to need); (3) output (productivity evaluation, 
staff productivity); (4) outcome (client satisfaction, family 
satisfaction, client outcomes); (5) output/outcome (cost 
effectiveness, cost/benefit analysis). Within each section, relevant 
literature is reviewed, specific techniques described, methodological 
concerns aired, and resource constraints noted. The final section 
discusses the implications of evaluation, including the uses of 
information to improve services, the role of "inside" versus 
"cutside" evaluation, and the potential future directions that 
evaluation is likely to take. A bibliography and appendix containing 
a client evaluation survey conclude the document. (JW) 
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Purpose of the Monograph 

During the past several years, it has become apparent that 
the provision of mental health services, especially tc persons 
with severe disabilities, is a complex enterprise that 
implicates a range of services and that requires r variety of 
professional and programmatic approaches. It is also 
increasingly clear that there is a need for an overarching 
ideology and set of principles to rationalize the system, to 
provide a continuing vision for the system, and to supply a 
basis for holding the system accountable. 

Advocates for community support programs are being 
challenged in many states to justify the efficacy and viability 
of the psychosocial rehabilitation concept through more targeted 
research. The increasingly visible problems of homeless 
individuals and the deteriorating conditions in many state 
psychiatric facilities are intensifying skepticism about recent 
reforms and are generating a potential reversion to more 
medically oriented programs. 

The past two decades have witnessed a tremendous growth in 
both the volume and the level of sophistication of research that 
addresses mental health systems and programs. Much of the 
resulting literature has focused on community mental health 
center outpatient programs and hospital based inpatient and 
outpatient programs. While some of these approaches are useful, 
there is a need for tools that focus on those issues that are 
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most relevant for planners and adininistratorf> of psychosocial 
rehabilitation programs, given the particular philosophical 
orientation of this complex program model. 

The discrete services evaluation efforts in the field to 
date, with a few notable exceptions, have not illuminated either 
the richness of psychosocial programs or the ef f ectivenes of 
such services in accomplishing their major aims , including 
community integration, client choice and autonomy, 
individualization, client participation, and participation in 
work. The development of evaluation mechanisms that more truly 
reflect the aspirations of programs for persons with chronic 
mental health problems ultimately will result in a clearer 
picture of such programs, helping administrators to communicate 
psychosocial rehabilitation principles to staff and clients 
alike. 

The following monograph advances a conceptual framework for 
the evaluation of mental health services provided by 
psychosocial rehabilitation programs, and more generally, for 
all mental health programs that purport to serve persons with 
chronic mental health problems. The framework takes into 
account the multifaceted character of such programs and the need 
to build an evaluation construct that recognizes the legitimate 
objectives of persons receiving services, their families, staff, 
legislators and the community. 

The monograph therefore, identifies those evaluation 
nechanisms that are consistent with the philosophy and 
programmatic aims of psychosocial rehabilitation programs. Some 
of the principles that govern such programs are: 
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• To assist people to regain those skills and capacities 
that are necessary for independent functioning; 

• To ensure multiple opportunities for client choice and 
participation in program planning; 

• To facilitate the integration of individuals in normal 
workr community and social activities; 

• To maximize individual diginity and personal 
development; 

• To provide services that are tailored to individual 
needs and circumstances ; 

• To involve families and other persons in the client's 
natural support system in the rehabilitation process; 

• To promote care in the least restrictive setting; 

a To provide a long term commitment to the continuing and 
changing needs of persons with chronic mental health 
problems ; 

• To maximize the use of a range of generic services; and 

• To advocate for the needs of persons with chronic 
mental illness. 

The audience for the following material is individuals who 

are interested in assessing the efficacy and quality of 

particular psychosocial rehabilitation programs. This includes 

program administrators, their evaluation staff, boards of 

directors, and clients and families. Because providers of 

psychosocial rehabilitation programs are not likely to have 

extensive resources to conduct highly sophisticated and 

intensive evaluations, the mechanisms proposed have been 

selected because of their relative ease of administration. 

The monogr^.ph is organized into three sections. The first 

section, the Introduction, provides a brief history of the 

psychosocial rehabilitation movement, proposes a conceptual 

er|c ^ 
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framework, and hi.ghlights existing evaluation publications which 
may be of use to program administrators. The next sections 
describe the components of a comprehensive evaluation stratgy 
and describes mechanisms aimed at the input, process, output and 
outcome dimensions of psychosocial rehabilitation programs. 
Within each section, relevant literature is reviewed, specific 
techniques are described, methodological concerns are aired, and 
resource constraints are noted. 

The final section discusses the implications of evaluation 
including the uses of information to improve services, the role 
of "inside" versus "outside" evaluation, and the potential 
future directions that evaluation is likely to take. 

Method 

In order to prepare the monograph, the authors carried out a 
wide-ranging search for exemplary approaches. First, a computer 
search wa*; made of relevant data bases. Second, NIMH staff were 
asked for their suggestions regarding significant but perhaps 
unpublished literature. Third, key informants in a variety of 
states were contacted to secure information. Fourth, 
announcements were placed in newsletters regarding the project, 
and fifth national mental health organizations were canvassed. 

The search for useful tools was also greatly facilitated by 
members of the project's advisory committee. In the initial 
phases of the project, the authors contacted a range of persons 
in the field of mental health and psychosocial rehabilitation 
with backgrounds as administrators, academics, providers, and 
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advocates. The advisory committee both assisted in identifying 
sources of information and provided a critical review of the 
final draft of the monograph. 
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I . INTRODUCTION 

Programs 

Psychosocial rehabilitation is a philosophical and 
programmatic approach to the care and treatment of persons with 
long-term, severe psychiatric disabilities. Its goals are to 
improve the quality of life of these individuals and to enable 
them to function in society as independently as possible. 
Underlying this approach are the assumotions that services 
should focus on individual strengths rather than disabilities 
(i.e., a "wellness model'*), that environmental supports are 
necessary to sustain people in their homes and communities 
(lASPRS, 1985), and that persons with serious mental health 
problems will respond to services that stress long-term 
rehabilitation rather than acute, medically-oriented care 
(Turner and Tenhoor, 157 S) , 

Within the rubric of psychosocial rehabilitation is a rich 
range of services: vocational training and job placement; 
training in skills of daily and community living; case 
management; social, recreational and educational services; and 
services that generate and sustain natural supports. The aim is 
to provide a comprehensive range of services to meet the range 
of potential client needs. Services are provided for brief or 
indefinite periods, depending on the intensity of the person's 
problem (lASPRS, 1985). 

Coordination of specialized and generic ser/ices is an 
important aspect of psychosocial rehabilitation. It is 
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facilitated by a de-emphasis on specialization, an informality 
in relationships between staff and clients, and a "functional" 
orientation (i^e., intervention directed at the development of 
skills linked to community integration) • Clients are encouraged 
to take part in service planning, delivery and evaluation 
(Laniol, 1980). In some ways, the psychosocial rehabilitation 
model is at odds with the traditional community mental health 
center approach that scresses the presence of particular 
services rather than the integration of services to enhance 
functioning. 

The psychosocial rehabilitation movement has been evolving 
since the late 1940 *s, when a group of formrr state mental 
hospital patients in New Yo.rk organized to found Fountain House, 
a social club (Pressing et al., 1983). For the first several 
decades of their existence, psychosocial rehabilitation services 
tended to operate in the private sector and to take on the 
character of the leadership of the particular program (Mosher 
and Keith, 1980) , 

Psychosocial rehabilitation services became more widespread 
in the late 1970 's stimulated by funding from the National 
Institute of Mental Health's Community Support Program (CSP) , 
Contributing to the development of CSP were: a recognition of 
the multi-dimensional service needs of people with severe 
disabilities, an acknowledgement of the failure of the 
conventional mental health system, including the community 
mental health center program, to respond to multiple needs, an 
openness to new approaches to treatment , and a recognition of 
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the pivotal role played by the states in the design and deliv€// 
of services. Although the community niental health movement of 
the 1960 's and 1970 's c itributed to a philosophy of treating 
all clients in the least restrictive environment and emphasized 
service coordination and the need for a range of services, 
clients with the most severe and persistent disorders often were 
excluded from the mental health system or were treated 
inappropriately. (For a detailed history of events and 
circumstances leading up to the CSP initiative, see Carling, 
1984a and Turner & Tenhoor, 1978). 

The CSP initiative called for a new conceptual approach to 
serving people with severe psychiatric disabilities rather than 
simply an expansion of services. Through the Community Support 
Program, NIMK contracted with state agencies to demonstrate and 
replicate strategies for providing comprehensive systems, or 
community support systems (CSSs) , at the local level. 
Guidelines for service system components were spelled out 
carefully. In addition to a psychosocial rehabilitation 
component, as defined above, there were outreach services, 
services to meet basic needs (e.g., housing, income support, 
health services, etc.), mental health services, 24-hour crisis 
services, and legal protection. The CSP contracted with states 
to improve local services and to run demonstration projects 
(Tessler and Goldman, 1982) . 

At its outset (1977-1978), CSP concentrated on non-elderly 
adults with chronic mental disabilities and the minority 
client. Housing and employment needs became integral parts of 

ERIC 
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the coEununity support concept. Administrative efforts to 
encourage interagency collaboration at all levels of government 
and to clarify financial responsibility were made as well. 
Means of involving consumers in planning were identified as a 
process goal (Carling, 1984b) . 

As CSP became established^ three clusters of underserved 
clients were identified : young adults , homeless people and 
minorities. Children's services were added as a separate but 
parallel program in 1984 with the 'development of the Children 
and Adolescent Service Support Program. More recently, it has 
become apparent that the elderly also are an underserved group 
(NIMH, 1986} . 

History of NIMH Evaluation Efforts in the Area of Psychosocial 
Rehabilitat • on 

CS? embodies much of the spirit of the psychosocial 

rehabilitation movement. While not all psychosocial 

rehabilitation programs are parts of Community Support Systems, 

psychosocial rehabilitation is, in fact, one of the community 

support program service components. The evaluation plan for 

the Community Support Program has followed a logical sequence of 

steps beginning with the award of an "evaluability" contract by 

NIMH to Macro Systems, Inc. in 1979. Conceived as the 

preliminary stage in an overall evaluation plan, that 

exploratory evaluation focused on broad system level issues. 

The project tasks included clarifying program goals at the 

federal, state, and local levels; Jiocumenting the program as it 

actually existed; and developing preliminary evaluation and 

management options (Macro Systems, 1980). 

Er|c 1 4 
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As a second stage in the CSP evaluation process, NIMH 
commissioned Human Services Research Institute to review the 
state of the art in needs assessment for the chronically 
mentally ill. Objectives of the project included the 
identification of available data on client characteristics and 
needs from CSP state reports and the mental health litero^ture; 
the examination of the comprehensiveness and compatibility of 
these data; the comparison of the relevant needs assessment 
findings from the state CSP reports and the literature; and the 
assessment of the major problems and limitations associated with 
the collection, presentation and interpretation of the available 
data (KSRI, 1979) • 

The third stage in the CSP evaluation strategy was a 1980 
NIMH contract to Macro Systems to develop a performance 
mesurement system and to conduct a short-term evaluation of 
CSP, Like the earlier Macro Systems exploratory evaluation, 
this project focused on system level issues and examined the 
implementation of federal intent at the state and local levels. 
The Macro Systems report gave the Comn^unity Support Program high 
marks for the development of a consistent and coherent 
conceptual framework for the conduct of programs, but stressed 
the importance of developing a body of information sufficient to 
support and justify the complex ideas and programmatic 
guidelines that underpin the CSP enterprise • Macro Systems 
pointed to the need to "isolate CSP effects by studying 
experimental and comparison groups," to "assess achievement of 
CSP intermediate and long range objectives," to "develop and 
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pilot test necessary instrumentation and methodolo^rical tools 
for collecting performance indicator data," and to determine the 
extent to which the •'quality of life of the target population 
has been improved** (Macro Systems)- 

In response to these recommendations, CSP evaluation 
proc*»eded on two fronts: the system level and the client level. 
In order to examine system level performance, KIMK awarded a 
contract to Professional Management Associates to conduct a 
field test of state and local self assessment instruments for 
community support programs (Stroul, Levine and Mulkern, 1985) . 
These instruments were field tested in 20 states and 94 local 
CSS areas. 

In order to meet the recommendation for a comprehensive 
client level data base capable of supporting both management and 
research needs, staff from three cooperating states — New York, 
Colorado, and Michigan collaborated with staff from NIMH to 
design the Uniform Client Data Instrument (UCDI). The 
instrument captures four areas of client related information: 
demographics, clinical history, adjustment and functioning, and 
service utilization. 

In 1979, NIMH awarded a contract to Market Facts Public 
Sector ' 3search Group for the purpose of pre-testing the UCDI. 
During the pre-test, data were collected on 1,471 chronically 
mentally ill individuals at fifteen CSP demonstration sites and 
three replication sites. The Market Facts report concluded that 
it was feasible to collect client data with the UCDI and that 
case managers were capable of supplying most of this 
information. 

ERLC 
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A more recent phase of NIMH's evaluation strategy was the 
award of a contract to Human Services Research Institute for a 
multi-state client follow up study (Mulkern, et al,, 1986). 
This research had four sub-studies: a cross sectional study, a 
longitudinal study^ a quality of life study, and a dropout 
study* This research presented a unique opportunity to develop, 
for the first time, a comprehensive body of information 
describing the characteristics, levels of functioning, and 
service utilization patterns of chronically mentally ill 
individuals . 

This set of research and evaluation projects has 
contributed to the continued improvement of services for persons 
with chronic mental illness. Projects have focused on the 
system level, the state level, local CSS areas, and the client 
level. With the exception of the client follow-up study 
mentioned above and a cost outcome study in progress, there has 
been littlfe research focused on the program or agency level. 
Since psychosocial rehabilitation centers play such a crucial 
role in CSP and embody so closely the philosophical 
underpinnings of the federal initiative, it makes sense to 
concentrate on these programs. This report is an attempt to 
fill a gap in CSP~sponsored research by providing 
recommendations concerning those evaluation research methods 
that are most appropriate and responsive to the mission and 
substance of psychosocial rehabilitation agencies. 

By helping to shape and clarify the nature of the 
psychosocial paradigm and the characteristics of the target 

•'^ i 7 
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population^ mental health evaluation research to date has laid 
the groundwork for more systematic analyses of community support 
services generally and psychosocial rehabilitation services 
specifically. The next set of tasks involves documentation of 
program results and program coherence. Such program level 
evaluation is necessary in order to maintain the dynamic quality 
of the community support program, to ensure a continuing 
exploration of new and innovative service techniques, and to 
build expanded public support for such services. 

Conceptual Framework 

As noted earlier, the task faced by the authors of this 
monograph was to provide those interested in evaluating 
psychosocial rehabilitation with a coherent, inexpensive and 
responsive set of strategies. In order to place these 
strategies within some context, it is important to advance a 
definition of program evaluation. According to Spaniel (1985) , 
program evaluation is: "a systematic, continuous process of 
providing information about the value of a program for purposes 
of decision-making" (p, 153), Through this definition, Spaniol 
stresses that evaluation should be an integral part of the 
ongoing operation of the agency; constitute a delj.berate and 
permanent feature of the organization; produce data that are 
valuable to decision makers; and make judgements about the 
•'goodness" or "badness" of agency practices. 

In other words, program evaluation differs from evaluation 
research in that the former is conducted as an adjunct to 
program administration and is justified by its utility to 



planning and continual program refinement. The latter is 
usually a discrete enterprise that is periodic and limited to r- 
narrow range of methodologies (Spaniol, 1985). Program 
evaluation is more sturdy and programmatic. 

In order to understand the relevance of available 
»ethodologies to particular dimensions of psychosocial 
rehabilitation programs, it is important to begin with a 
conceptual framework. The simplest way to represent the aspects 
of programs that form the targets or subjects of evaluation is 
reflected in the flow chart on the following page. 

Taking the above general overview as a guide, the next step 
is to determine those aspects of psychosocial rehabilitation 
programs that are unique and in line with the particular mission 
and purpose of such programs. 

Inputs . Those inputs with particular relevance to 
psychosocial rehabilitat^'.cn include the agency's mission, the 
level of conpetetence cf staff, the financial condition of the 
agency, and the needs of the clients. 

• Mission 

A major determinant or input into psychosocial 
rehabilitation programs is the agency's mission. It is this 
mission statement that sets the tone for service delivery and 
that alerts staff, clients, friends and families, and the 
community to the rehabilitative and reintegration objectives of 
the program. 

• Client Characteristics and Needs 

The target groups served by psychosocial rehabilitation 
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COMPONE TS OF EVALUATION FRAMEWORK 



PROCESS 



INPUTS 



The resources 
available to the 
progrcun and the 
nature and needs 
of clients. 



A 



The inter- 
ion between 
the client and 
the organization 
providing ser- 
vice, and the ad- 
ministrative and 
support activi- 
ties • 



OUTPUTS 



OUTCOMES 



What 
happens 
to the 
clients as an 
outgrowth 
of the 
service . 



The products of 
the service pro- 
cess as a func- 
tion of cost 
and staff effi- 
ciency and pro- 
ductivity. 
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programs tend to be composed of people with complex problems and 
chronic needs for a variety of mental health and generic 
services (e.g., housing, employment, income, etc.). It is 
encumbent on such programs to insure access to those most in 
need and to be instrumental in providing and/or brokering needed 
services . 

• Staff Competency and Orientation 

The success of psychosocial rehabilitation programs depends 
almost entirely on the extent to which program staff have 
internalized the mission and values of the agency, have 
developed a respect for the individuality and dignity of the 
clients they serve, and have achieved competency in the various 
service approaches that comprise the program. Therefore, any 
evaluation of inputs adopted by a psychosocial rehabilitation 
program must include a means of assessing staff competencies. 
• Financial Condition 

Psychosocial rehabilitation programs are subject to the 
vagaries of public funding and as such are vulnerable to 
destablizing cutbacks and uneven reimbursements. Understanding 
the agency's financial condition on a regular basis is crucial 
if program administrators and board members are to make informed 
resource allocation decisions. 

Process . Those aspects of the process of service delivery 
that are an important focus for psychosocial rehabilitation 
evaluators include the quality of service delivery and its 
relationship to accepted psychosocial rehabilitation practices, 
the extent of community connectedness and the extent to which 
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the program meets client need, 
# Prograiu Practices 

As the field of psychosocial rehabilitation has evolved over 
the past 20 years, the sense of what constitutes "best practice" 
has become increasingly solidified. Through consensus in the 
field and standards, such as those developed by the Council on 
Accreditation of Rehabilitation Facilities, there are principles 
of psychosocial rehabilitation that can be used to judge the 
quality 'of service delivery. Such principles include the 
development of independent living skills, the integration of 
clients into the community, the maximization of natural 
supports, and the inclusion of clients in decisions regarding 
their service plans, 

• Program Connectedness 

Because persons with chronic mental health problems have 
multiple and complex needs, psychosocial rehabilitation programs 
must provide services as well as broker other services — 
especially those services that meet basic human needs for food, 
clothing, and shelter. The extent of interagency coordination 
is therefore an important element in any evaluation of the 
service process, 

• Match to Need 

Again, because of the complexity of service needs among 
clients of psychosocial rehabilitation programs, it is clearly 
advisable to review continually the agency's ability to meet the 
needs of clients as initially anticipated in client plans. 

Outputs , The two major program outputs that should be of 
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concern to psychosocial rehabilitation programs are financial 
productivity / and staff productivity. 
m Financial Productivity 

The standard measure of human service efficiency is the cost 
per client served. In psychosocial rehabilitation programs, 
this ratio is usually understood in conjunction with an 
assessment of the organizational, programmatic and client 
variables which contribute to or detract from the agency's 
productivity. 

• Staff Productivity 

Conventionally, staff productivity means the number of 
products completed by staff or the number of objectives 
achieved. In the psychosocial rehabilitation field, 
productivity is related to the ability of staff to both complete 
discrete tasks and to posi'cively affect the lives and 
circumstances of their clients. 

Outcomes. This dimension of any evaluation scheme focuses 
on program results. Since psychosocial rehabilitation programs 
have multiple ends, any assessment of outcomes should include 
client satisfaction, family satisfaction and return to 
independent functioning. 

# Client Satisfaction 

One of the significant features of psychosocial 
rehabilitation programs is the client's relationship to the 
services. According to psychosocial rehabilitation principles, 
clients not only use services, but also are encouraged to 
participate in the making of choices about their care and 
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support. They are both recipients and participants in the 
planning and evaluation of their services. Thus, any evaluation 
should take into account the objectives of client participation 
and autonomy* Clients are also primary sources of information 
regarding the integrity of the agency *s policies vis-a-vis 
psychosocial rehabilitation principles (e.g., being treated with 
dignity), and the efficacy of the agency *s program (e.g., extent 
of community integration and return to functioning) . 

♦ Family Satisfaction . One of the major actors in a 
comprehensive program of community support for persons with 
chronic mental health problems is the family. The stereotype of 
persons with chronic mental illness is of a loner living in a 
single room occupancy hotel or in a boarding home; however, 
statistics indicate that large numbers of people live at home 
with their families (Ashbaugh, 1933) . Until quite recently, 
programs designed to support such families were virtually 
non-existant. Counselling did not necessarily contribute to the 
family *s capacity to cope with the problems of their mentally 
ill family member; rather, it concentrated on the pathology of 
relationships within the family unit. 

In order to ensure that their needs are incorporated into 
comprehensive psychosocial rehabilitation services, family 
perspectives on the dtslivery of services — their opinions and 
experiences - should be canvassed as an ongoing part of any 
evaluation mechanism. In assessing the level of satisfaction of 
families, it should be kept in mind that their perceptions of 
service success may vary and in some instances be at odds with 
those of the clients as well as those of the staff. 



. -19- 

# Client Outcomes > The major objective of psychosocial 
rehabilitation programs, unlike conventional mental health 
services, is not "cure" per se , but the achievement of a level 
of functioning that is as normal as possible. Such achievement 
may come through habilitation (i.e., the initial establishment 
of skills) or rehabilitation (i.e., the resuscitation of skills 
that have been lost) . This includes functioning in a productive 
work setting and participating in an integrated set of social 
and leisure time activities. Most of the scales which have been 
developed to measure outcomes in mental health ha . focused on 
the reduction of discrete symptomology . While this is clearly 
an important activity, it does not tell us whether or not such a 
reduction in symptomology has resulted in increased functioning 
in the community, accelerated social integration, expanded 
personal autonomy, or enhanced quality of life. 

The measurement of outcomes in psychosocial rehabilitation 
programs is complex. The fact that clients are enrolled in 
multiple service programs offered through a variety of different 
service systems makes it difficult to attribute observed changes 
in functioning to the program in question. Even with the 
development of outcome measures that are reliable, valid and 
sensitive to change, the problem of causality remains. 

Further, the importance of individualized planning in 
psychosocial rehabilitation programs neccessitates outcome 
measures that are sensitive to the very personal nature of 
"success" for each client. Concurrently long-term and episodfr, 
the nature of the typical client's disabilities places furtner 
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demands on such measures. Chance occurs slowly and frequently in 
a non-linear fashion and outcome measures need to be sensitive 
to smaller scale gains. 

Despite these difficulties, program planners and 
administrators cannot ignore the fact that client ou)- ome is the 
bottom line by which program success ultimately is measured. 
Flawed as it may he, outcome research at the agency level is an 
integral part of program management. 

Outputs/Outcome . The final component of the conceptual 
framework combines outputs — cost per unit of service — with 
client outcomes. This form of assessment is characterized as 
cost effectiveness analysis and requires that the other two 
program dimensions have been addressed. 



Existing ^ luation Resources 

It is not the intent of this manuscript to describe general 
methods of evaluation research; this task has already been 
accomplished to a large extent in several reports sponsored by 
the National Institute of Mental Health. The first of these is 
titled Resource Materials for Communitv Mental Health Program 
E valuation (Hargreaves, Atkisson, and Sorenson, 1977), Intended 
to strengthen the evaluation capabilities of local agencies, 
this monograph contains articles concerning evaluation research 
and discussions of selected techniques. It covers a va]:iety cf 
issues : 

• Elements of program evaluation; 

• Needs assessment and planning; 

• Management Information systems; 

o 



# Evaluation of the effectiveness of services (this 
section reproduces selected ins truKients for use in 
CMHCs); and 

# An extensive bibliography that categorizes the 
literature under four headings including systems 
management, client utilization, outcome, and 
community iiupact . 

The second NIMH document is titled, A Working Manual of 

S imple Program Evaluation Techniques for Community Mental Health 

Centers (Hagedorn, Beck, Neubert, and Werlin, 1976) • This 

monograph contains general evaluation material including 

discussions of the nature of evaluation research jind ways in 

w.^ich to link evaluation research to program change. It also 

describes the administration of program evaluation in a CMHC — 

developing an annual plan, choosing topics, designing the 

evaluation, conducting the research, and maximizing the 

utilization of the results. Technical sections include methods 

for. research on needs assessment, patterns of use, cost 

analysis, impacts and outcomes, cost outcome and 

cost-effectiveness, the effectiveness of consultation and 

education, quality assessment, and approaches to citizen review. 

A third monograph, which might be of use, is Citizen 

Evaluation in Practice (Bradley, Allard, and Mulkern, 1984}. 

Containing descriptions of citizen involvement in a variety of 

evaluation studies, this casebook orgar zes evaluation examples 

according to five perspectives: the citizen-evaluator * s 

position vis a vis the program being evaluated (e.g., internal 

or external); the evaluator*s level of involvement; the scope of 

the evaluation; the topics to be studied; and the methods to be 
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used. Each case is described in terms of: 

• The type of organization conducting the research; 

• Characteristics of the individuals ::onducting the 
research (e*g* professional researchers, lay persons, 
clients , etc • ) ; 

• Reasons for the evaluation or monitoring; 

• Target of the evaluation or monitoring; 

• Problems or issues evaluated or monitored; 

• Techniques used; 

• Findings of the evaluation or monitoring activity; 
Recommendations ; 

• Steps taken to ensure implementation ; 

• Extent of implementation; 

• Special barriers to, or supports for, the evaluation; 
and 

• Resources and costs of the evaluation or monitoring. 
The usefulness of this monograph lies in its coverage of 

topics that are frequently omitted in rost of the professional 
literature on evaluation research, specifically the resources 
required for the research, strategies for disseminating the 
results, and ways of maximizing the research's impact. 

The final NIMH supported publication is titled. Evaluation 
in Practice: A Sourcebook of Program Evaluatio n Studies from 
Mental Health Care Svstems in the United States (Lands berg, 
Neigher, Hammer, Windle and Woy, 1979) . The Sourcebook includes 
a wide range of cases that highlight multiple approaches to 
program evaluation in both discrete mental health programs and 
systems of services. The areas covered include needs 
assessment; patterns of service use; acceptability, 

o 2 8 
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availability, accessibility and awareness; client outcomes- 
community impact; indirect services; cost, cost outcome, and 
cost effectiveness; quality assurance; and mental health program 
evaluation in the Year 2000. This book is still available from 
NIMH, and is an enormously useful reference for basic evaluation 
practices. 
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II. INPUT EVALUATION 
Recognizing that the development of a program evaluation 
strategy may o£ necessity occur in stages — given the need to 
identify resources and to introduce procedures gradually — the 
'most logical point to begin is with the input dimension of the , 
program. It is here that the administrator and others concerned 
about the viability of the program can ensure that the "raw 
material'* which the program has to work with is adequate, that 
tne clients served are appropriate and that client needs have 
been defined. 



Aaencv Mission 

The first +-.ask in the evaluation of inputs to psychosocial 
rehabilitation programs is to assess the agency's mission in an 
attempt to determine whether it is consistent with the 
principles outlined in Section I above. There are several ways 
in which the presence and implementation of a mission can be 
judged: 

1. Review the agency's mission statement to determine 
whether it addresses the major psychosocial 
rehabilitation principles. 

2. Determine the extent to which the mission statement is 
made avail ble to staff, board members, agency clients, 
and family members. 

3. Assess how frequently the mission statement is 
discussed with staff and board members and whether the 
objectives are reviewed periodically to ensure that the 
mission remains relevant and up to date. 

4* Review personnel policies, in-service training 

materials and job descriptions to ensure that the 
agency's mission is inculcated in initial and ongoing 
staff orientations • 



5. Review reports to the board, agency brochures and othe: 
formal agency documents to ensure consistency with 
.vacrency mission in program descriptions. 
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6. Review service planning protocols to determine the 
level of client involvement and choice. 

7. Review agency procedures to determine how often the 
views of clients and family members are canvassed to 
ascertain the level of satisfaction with agency 
programs . 

8. Assess the extent to which skills training and 
development is directly relevant to increases in 
independent client functioning and community 
integration. 

There are some tools available that may be helpful to 
program administrators in assessing their agency's mission. One 
is the set of exercises in the Community Systems Workbook 
(DeSisto & Ridgway, 1986). The workbook is organized according 
to service components that are related directly to psychosocial 
rehabilitation principles. Each section begins with a 
description of the service area and a goal/mission statement and 
continues with a list of discussion questions. The questions 
attempt to operationalize psychosocial program concepts and 
specific service approaches. Exhibit 1, which covers the next 
three pages, presents the section from the workbook on Housing, 
The entire scope cf service components covered are: 

1. Integrating Services 

Outreach and Case Finding 
Comprehensive Individualized Assessment 
Comprehensive Individualized Planning 
Facilitating Linkages, Coordination, Advocacy 
Modifying and Creating Resources and Supports 
Transportation 

Monitoring, Evaluating, Reassessing, Revising 
Meeting Special Needs 

. 2. Basic Supports 

Income Supports 
Housing 

Supportive Care and Supervision 
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miBIT 1 
ASSESSMENT OF AGENCY MISSION 



Many psychiatrically disabled persons lack access to decent^ safe, 
and affordable housing due to their inadequate income and the 
generally low availability of subsidized units and lov income housing 
stock, as veil as discrimination due to stigma. In addition, the 
lack of an adeijuate support network and repeated acute episodes of 
illness often result in the loss of housing. Without a stable living 
environment, treatment and rehabilitation have a very low chance of 
success. 

GOAL - TO ENSURE THAT EACH PSYCHIATRICALLY DISABLED PERSON HAS ACCESS 
TO DECENT AFFORDABLE HOUSING THAT IS MATCHED VITH HIS OR HER SKILLS, 
RESOURCES, AND GOALS. 



HOUSING FUNCTIONS 

1. Does the agency/service area actively help each client to work to 
achieve a decent living environment? 

^NO 

^YES 

2. Does the agency/service area provide a variety of services to 
link psychiatrically disabled persons to housing? 

Help in locating housing 



Maintaining lists of rental housing 
Recruiting landlords 



Linkage to public housing and rental assistance 

3. Does the agency/service area provide the resources 
patients/clients need to secure and maintain housing? 

Subsidized Housing/Housing Supports 

Subsidized Security Deposits/Loans 

Assuring Landlord Rent 

Sublet Leased Apartments 

Provide References to Clients 



Household Supplies, Equipment and Furnishings 
Moving Patient/Client Belongings, 
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4. Does the agency/service area vork to protect housing/placement 
short-^term hospitalization? 



5. Does the agency/service area have temporary housing available 
for crisis/respite needs? 



6. Are specialized residential programs for special populations 
available in the agency/service area when needed? 



7. Are there a variety of housing options available? 
Roommate matching/cooperative apartments? 



Supported independent living housing options?^ 



Non-treatment residential options linked to intensive programs 
and supports? 



Substitute family environments? 



Non-institutional small group living?^ 



Is there support, education, and consultation available to home 
care providers (natural families, foster family, spouse, 
significant other, boarding care providers) ?_ 



Is there back-up support to landlords? 



Has the agency/service area undertaken community or systems 
advocacy in order to: 

increase the availability of subsidized housing?, 
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end housing discrimination? 

reduce stiffma and increase coinisunity acceptance? 



10. Are there efforts to familiarize patients/clients vith their nev 
living environment and community vhen a move occurs? 



11. Are patients/clients supported in undertaking the activities 
necessary to find a living situation? 

using classifieds/finding unadvertised living situations 

getting utilities hooked up 

finding and choosing roommates 



12. Does the agency aid the person vith the tasks necessary for them to 
get settled in a new environment? 

moving and decorating 

negotiating rules and responsibilities vith roommates 

budgeting/ shopping 

cooking/cleaning 

safety. 

13. Does the agency/service area aid the patient/client in maintaining 
the living situation? 



landlord tenant relations 



getting along vith roommates/others 



14. Does the agency/service area assess and aid the patient/client in 
the transfer of functional skills vhen he or she enters a new 
living situation? 



Source: DeSisto, M.J. & Ridgeway, p,. Community Systems Workbook , 1986. 
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3. Treatment Services 

Psychiatric Assessment and Medication 

Psychotherapeutic Services 

Crisis Services 

Drug and Alcohol Services 

Health Care Services 

4 . Rehabilitation Services 

Skill development (daily living, social skills, 
vocational rehabilitation) 

Social Rehabilitation (social support network 
development , social and recreational activities , social 
stigma reduction) . 

The questions in the workbook can be administered as a self 

assessment tool by agency staff, by board members in interviews 

with staff, or by agency clients also through interviews with 

staff. 

A second guide. Program Ane.lvsis of Service Systems (PASS, 
Wolf ensberger and Glenn, 1976), was designed as a monitoring 
tool in the field of mental retardation. PASS is one of the 
only scaled survey tools that attempts to operationalize an 
ideology — referred to as normalization — which is premised on 
integration and the reduction of stigma. While PASS was written 
primarily for a different field, the principles it espouses are 
universal; ^hey force an agency to look at the extent to which 
its programs are individualized, dignity-enhancing, age and 
culturally relevant, and client-centered. Administrators should 
review P^SS and pick out those items that are the most relevant 
to their own service approaches and settings. 

Regardless of the method used, a continual review of agency 
mission and its integration into day to day procedures is a 
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crucial first step in ensuring the integrity of psychosocial 
rehabilitation services . 



Client Characteristics and Needs 

Clients enter psychosocial rehabilitation programs (or any 
other service program, for that matter) with an almost infinite 
variety of characteristics and service needs. The documentation 
of client profiles and service needs does not constitute program 
evaluation per se. However/ it is a necessary prerequisite to a 
full understanding of organizational inputs. Systematic 
reporting of the types of clients entering the program is 
essential both for service planning and for understanding 
variations in treatment impact. 

Instrumentation and Approach . Most of the information 
needed to construct client profiles will be available on agency 
intake forms. It is possible to collect huge amounts of 
information describing clients coming into programs. In many 
cases, however, only a few key data elements are used 



rout 



inely. These include: 
age 
race 
sex 

education 

current living situation 
income sources 
primary diagnosis 
secondary diagnosis 

previous use of mental health services (including 
inpatient services) 
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Information on service needs can be collected at the time 
that the client's treatment plan is developed. In order to 
assure comparability across cases, this type of information 
should be recorded on a standard form. In addition, all persons 
working with clients to develop treatment plans should be 
trained so that the same service definitions are applied in alx 
cases • 

Exhibit 2 is an example of a service record that has been 
developed by H. Stephen Leff and colleagues at Human Services 
Research Institute.* Columns one and three are relevant in this 
context. Columns two and four will be addressed in a later 
section on "Match to Need". 

The left column contains a service taxonomy. The taxonomy 
should reflect local service patterns and nomenclature. Case 
managers or service planners indicate whether or not each 
service is needed by a specific client. If more precision is 
desired and if the agency feels capable of analyzing further 
data^ case managers can also report the units of service needed 
(column three on this particular form) . 

Since service needs change routinely, provision should be 
made for this information to be updated periodically. If the 
agency requires that the treatment plan be revised according 

* This instrument is still under development by Leff, et al. at 
Human Services Research Institute. Evaluators interested in 
using the form should contact HSRI in order to obtain a current 
version. 
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SERVICE HECOFD 



SEKVEE OKNENT 


Type 
liut 




ISSHENHAL 






day 


a,1 


b. Ccnninity festal 


day 


b.1 


Crisis Beds 


day 


C.1 


d. Respite feds 


day 


d.1 


e. E^sic Rasidential 


day 


e.1 


f . Sf^RT-iPlty Residential 


day 


f.1 


g. Intensiva Rssici=rri;^l 


dsy 


g.1 


Fester CarB 


day 


h.1 


i. Fsnily Sifcsicy 


dsy 


i.1 


j. Irri2;:en3£i"it Livirg 


day 





(1) 

Obck 71 
Ssrdce 
In Qient 
Service 
Plan 



TEEXMNT 



k. Brsrgency AssessiEnt 


hairs 


k.1 


1. M±ile Ib^tiTEnt Tesra 


days 


1.1 


m. Ccurt^irg 


hars 


m,1 


n. Faraly Iraatnsnt 


hojrs 


n.1 


b. Substancs Abuse TkOatpt 


hxzs 


O.I 


p. bfed ^bintenaxe 


hojrs 


P-V 


q. Efey jteat:T^.t G hr) slots 




Activities G hr) slots 


r.1 



s. Vbc/EdAsssssTEnt (4 hr) slots s.1 

t. STsltsred Vtd^^tv (4 hr) slots t.1 

u. Siarctsd Wbck & IE? (4 hr) slots l,1 

V. J±) Firdirg/ESvel cpudi it hxrs v. 1 

w. Edraticcal Services hxrs w.1 

SEKET 

X. Case hfer^«7Ent h^jrs x.1 

y. liop-in Center hears y.1 

z- feeral axpsrt hajrs z.1 

aa- PrDtectiaV/cvxargr bars se.1 

*OSr CF Ka^S2S CmtN 4 

01 Service vas ret avail^le 
C2 Service has insufficient cspazity 
CB Client ves redsed for bei^victci 
reasons .v 



(2) 
#Uu:s 
Araally 

lb Client 
Per Mn* 



- a.2 
.b.2 
C.2. 

d. 2_ 

e. 2_ 

f. 2_ 

g. 2_ 

h. 2_ 

i. 2_ 
j-2_ 



k.2 

1.2 

tn-2 

n.2. 

o.2_ 

F-2. 
q.2. 
r.2_ 

s.2_ 
t.2_ 
u.2_ 
v.2_ 
•W.2 



X.2 

y-2, 

Z.2. 

SS.2 



J) 

Services (+ cr -) 
Qient SxxM 
Ideally R83si\^ 
W.th EsdnECs Of 
N-rrbsr Cf Lhits 
Fer ftnth 



a. 3 

b. 3. 
C.3, 
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e. 3_ 

g. 3. 

h. 3_ 
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j-3_ 



k.3 

1.3 

m.3 

n.3. 

0.3 

P-3. 

q-3. 

r.3_ 

s.3_ 
t.3_ 
u.3_ 
v.3_ 
W.3 



X.3 

y-3, 

Z.3 
S.3 



W. 0 

feasors Vhy 
Services Lt 
Cblum 3 ■ 
Nx Frcvided 0 
Fe«er/btre Ihit 
ThanCblum^ 
(Ihdicsce W 
That itoolv) 



a. 4, 

b. 4 

c. 4_ 

d. 4_ 

e. 4_ 
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h. 4_ 

i. 4_ 



k.4 

1.4 

tn.4_ 

n.4_ 

0.4 



r.4 



S.4 
t.4_ 
u.4_ 
V.4 



W.4 



X.4 

z.4_ 
sa.4 



• ■■^^■ ' ■'~t.'.^d.J,i.A}.XX/.*.k>J.iJ,tJJJJ.kJ./.Xi^X^:J<i.XJ>. ^ ^^^ 



0^ Inability to pay 
C5 Client refised service 
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06 Lsrs-^e cr culnrcl barrier 
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to a specific schedule, this would be an ideal time to update 



reason, cannot be implemented) , then the service record should 
be updated every three to six months. 

Staff Competence 

Background. A variety of tools to assure the competence 
of staff working with persons with chronic mental disabilities 
have been developed over the years at the federal, state and 
agency level. Certification programs, such as the criteria for 
psychiatric technicians in California, have formed the basis for 
ensuring the skill level of particular categories of direct care 
staff. While these approaches test the acquisition of 
particular techniques such as medication monitoring, behavior 
management, and skills teaching, they are not necessarily set 
within a framework that is relevant to psychosocial 
rehabilitation programs. Recently, two approaches to ensuring 
staff comptency within such settings have been developed. 

The first. Assessing and Improving the Performance of 
Psychosocial Rehabilitation Staff (Friday and McPheeters, 1985), 
was funded through a grant from the NIMH Center for State Human 
Resource Development. The authors break down competence in 
psychosocial rehabilitation into five categories: 1) 
informational, 2) intellectual, 3) interpersonal ,4) 
intrapersonal, and 5) interventional. Within these categories, 
this publication identifies a number of specific competencies 
necessary for workers to perform at optimum level. With these 
competencies identified, the report describes possible measures 
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Assxunptions and Methodological Issues . It is through the 
development of competency assessments that the agency 
communicates its expectations and values. Therefore, this 
aspect of an agency's evaluation is essential to ensuring the 
coherence of the agency's program and its capacity to meet its 
objectives. In line with this notion, Friday and McPheeters 
assert that an agency first must think through those skills that 
can be associated reasonably with positive client outr^omes. 
They suggest that each staff person's job description should 
include intermediate measures of outcomes for the clients whom 
the workers are rehabilitating. Exhibit 3 lists some sample 
measures of outcomes from job descriptions. 

Additionally, Friday and McPheeters outline a simplified 
form of "employee credentialing" as a means of ensuring and 
evaluating competence. The steps involved in developing such a 
system are described in Exhibit 4. 

The key assumption underlying the Friday and McPheeters 
approach is that the evaluation of competence is not merely a 
means for setting salaries and determining promotions; rather it 
is a means of helping workers "achieve more effective and 
productive performance, and thus greater job satisfaction." In 
other words, the assessment of staff competence should be a 
means for communicating the agency's mission and values, 
identifying those skills most likely to accomplish the mission, 
and supplying remedial training whenever indicated The process 
is greatly enhanced if it is seen by staff as a way of 
continually improving their abilities rather than as a way of 

o 4 0 
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to assess and improve the competence of psychosocial 
rehabilitation workers. The document also cites the use of 
agency indicators and performance reviews by clients as means by 
which staff performance can be assessed. The appendix to the 
study includes a list of competencies and a questionnaii e . 

The second assessment method reviewed is part of ^he 
Psychiatric Rehabilitation Agency Assessment Program (Center for 
Rehabilitation Research and Training in Mental Health, 
undated) . The assessment is in some ways akin to an 
accreditation. Paid for by the psychosocial rehabilitation 
program, the review includes three parts; system of 
rehabilitative environments, rehabilitation program components, 
and -staff practice components. The latter is defined as 
f ol,'' ows : 

The knowledge and attitudes of staff practitioners in each 
environment about psychiatric rehabilitation and the 
psychiatrically disabled. 

The current level of psychiatric rehabilitation skill 
demonstrated by the staff. {p. 4) 

The methods used by the Research and Training Center to 
ascertain the nature of staff attitudes and the level of 
competence include a review of personnel data through group 
intereviews, written performance measures, staff self assessment 
of rehabilitation knowledge, and direct observations of 
staff /client interaction. 

Both of these approaches involve a sensitivity to the 
specific mission of psychosocial reh.;bilitation programs and a 
concern for the competencies required of staff to assist persons 
with chronic mental illness co return to normal community 
living. 

ERJC 
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Approach. The manual, Assessing and Imp rovi ng the 
Performance of Psychosoc ial Rehabilitation Staff by Friday and 
McPheeters (1985) , offers the best guide for the assessment of 
staff competence. Their work distinguishes such assessments in 
this field from standard competencies measures. Specifically, 
they note: 

[A] definition of competence for mental health workers 
:.ncludes three concepts that lie beyond the usual concepts 
of proficiency in knowledge and skills: 

the concept that attitudes and values of the worker 
are critical to competence; 

— the concept that competence requires personal 
interaction with clients; 

the concept that compeL^ence requires effective outcomes 
for clients, not just prcficient performance by workers, 
(p. 26) 

This analysis is in keeping with the theme of this monograph — 
that discrete aspects of psychosocial rehabilitation programs 
cannot be evaluated without returning to the values and goals of 
such programs • 

Friday and McPheeters describe several ways of assessing 
competence: certification and credentialing; supervisor's case 
conferences; supervisor's performance evaluations; peer 
conferences and evaluations; client ratings of staff competence 
and case record audits; and extrapolating from agency indicators 
(e.g., client drop out rates). They conclude that "the more 
effective approaches are those that most clearly define the 
tasks arid competence levels required for each worker in contrast 
to those approaches in which the expectations are more global 
and nonspecific." 



42 



-37- 
EXHIBIT 3 

OUTCOKE MEASURES FOR JOB PERFORMANCE 

!• Assesses the psychosocial functioning of cl: >nts and records these 
data in the clients' clinical records. 

Supervisors may then evaluate the quality of the worker's psychosocial 
assessments of clients and whether these were completed in a timely 
fashion and in the numbers expected. 

2* Plans individuali zed T?rocTram of incremental goals for and vith clients 
and records these in the clinical record. 

Supervisors may then evaluate the quality of these plans and their 
timeliness. 

3. Models a nd teaches clients the needed behaviors identified in the 
plan. 

Supervisors can evaluate both activity records and documentation in 
the clinical records to assure that these activities are being carried 
out. 

^- Structures opportun ities and assigns clients so that th^v ran 

experience appropriate behaviors according to the treatment plan. 

Tlie worker's success in performing these activities can be evaluated 
by reviewing the clinical records and client activity reports. 

^* Motivates^ counsel s, and supervises clients to participate in the 
program as structured. 

This item is more difficult to evaluate. Client contacts regarding 
these activities should be documented in the clinical records. 

5- Challenges inappro priate behaviors of clients and takes actions to 
correct them. 

This activity should he documented in the clinical record, and also by 
observing the incidence of inappropriate behaviors. 

Reassesses psychos ocial progress of clients and revises plans or 
refers clients to commu nity resources, and records these plans and 
activities. 

This step may be evaluated by observing the plan revisions in the 
clinical record. It may also be measured by client referrals to 
^comnunity resources, and termination from the program or unit of the 
program after clients have attained the outcomes defined in the plan. 



Source: Friday, J.C. & McPheeters, H.L., Assessing and Improving the 
Performance of Psychosocial Rehabilitation Staff , 1985. 
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EXHIBIT 4 
STAFF CREDENTIALING 



1. Reviev all job descriptions to assure that they represent the full 
scope of job duties that are expected of each vorker> 

2. Develop a list of competencies {i_.e. the knovledcre, skills, attitudes, 
and values) required for each position, 

3* Develop an orientation program for each emt>ljyee to every competency 
and job procedure. 

This should be completed within the first two months of employment 
through structured training programs or through individualized 
supervision, reading lists, or other means. 

4- Identify a staff person to - heck each new employee's ability to 

perform on all of the competen cy items that have been deemed essential 
for his/her iob> 

When the new employee has sucr'^ssf ully met these standards, he or she 
is "credentialed" by the agency, usually after two to three months, 
demonstrati.ng initial assurance that the worker has the capacity to 
perform adequately. At this point, the worker is given permanent 
status. 

^* Evaluate each worker at periodic internals (e.cr.. everv six months) on 
the duties of his/her job description. 

Decisions are made about what deficiencies exist and how they may best 
be remedied. The worker is than recredentialed if performance is 
satisfactory. This audit provides the occasion to reevaluate the job 
description, the overall agency procedures, and the functioning of the 
program's teams. The worker and the supervisor can jointly explore 
ways to make the entire unit more effective and productive. 



Source: Friday, J.C. & McPheeters, E.L., Assessing and Improvinc the 
Performance of Psychosocial Rehabilitation Staff . 1985. 
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identifying and punishing faults. This former attitude can be 
reinforced by inducing staff in the discussions of job 
descriptions and competency measures. 

Issues of staff productivity and the development of criteria 
to assess the supportiveness of the agency's environment will be 
treated in the section on output evaluation* 

Instruments . The development of instruments to assess 
competency must be preceded by an analysis of skill components. 
As noted above, Friday and McPheeters break down required 
competencies into five parts: informational, intellectual, 
interventional, intrapersonal, and interpersonal. They use 
these categories to describe particular skills and knowledge. 
Exhibit 5 outlines the skills suggested for interpersonal 
competence. They then recommend that once general competencies 
have been established, separate skills rankings should be 
determined for each job description, in order to tailor 
competency assessment to each job category. 

Implementation and Analvsis . Arriving at personnel 
competencies for an agency should involve not just the 
administrative staff, but all staff affected. In this way, the 
process is made less threatening and multiple professional 
viewpoints are reflected. Insofar as monitoring competence, a 
number of methods are suggested in the Overview above. 
Competencies should be determined at least through case and 
supervisory review as well as through client satisfaction 
surveys. staff reviews should occur every six months. 
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SKILLS REQUIRED FOR INTERPERSONAL COKPETENCE 



III. INTERPERSONAL 

These are the abilities, such as genuiness, empathy, warmth, and the 
ability to confront realisitically, that were described by Carkhuff 
and others • It also includes such interpersonal characterisics as 
courtesy and positive regard. 

III.-l. Ability to tolerate low profile — realization that one may not 
always be rewarded or recognized for their activities. 

III. -2. Ability to feel good about small changes and appreciate 

positive change in client co-worker — no matter how small 

III. -3. Be curious 

a. Be inquisitive, especially in client's concerns 

b. Active desire to investigate and learn 

III.-'4. Empathy and sensitivity — have some understanding of the 
feelings, thoughts and experiences of others, also the 
ability to sense the feelings and communicate to others 
(i.e., client, co-workers, agency personnel, etc.), 
sensitive to the needs of the consumer 

III. -5. Have integrity 

a. Honesty — doing what you say you are going to do (i.e., 
follow through) 

b. Not doing things that will compromise self or clients 

III. -6. Genuineness 

c. Authentic, real honest 

III. -7. Non-possesive warmth — ability to establish a caring but not 
a consuming relationship with the other person 

III. 8. Have positive regard for clients 

a. Treat clients as individuals without belittling them or 
making them feel inferior 

b. Like and respect clients 

c. Do not demean or patronize clients 

d. Respect clients dignity and privacy 

e. Avoid "labeling" people ^ \th sterotypes or derogatory 
terminology 

f . Avoid judging persons — consider only their behavior 




Source: Friday, J.C. & McPheeters, H.L., Assessing and Improving the 
Perfomance of Psychosocial Rehabilitation Staff, 1385. 
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Financial Condition 

Background . An important area of concern for psychosocial 
rehabilitation programs is the financial stability of such 
agencies. Given that many of the services provided under the 
rubric of pyschosocial rehabilitation are not reimbursable under 
Medicaid and other entitlement programs, agencies are more 
vulnerable to the ebb and flow of public funding. For this 
reason, evaluation approaches that assist the agency to assess 
current financial status are of particular importance. A wealth 
of references for evaluating an agency's financial condition can 
be found in business administration and to a lesser extent in 
public administration literature; however, only one was specific 
to psychosocial rehabilitation agencies: The Mental Health 
Statistics Improvement Program (MHSIP) Financial Data Task Force 
(April, 1986) on financial indices to be included as part of the 
MHSIP Mental Health Data Standards. These information standards 
are intended to lead state agencies to collect reasonably 
uniform data allowing cross-organizational and cross-state 
comparisons • Such comparisons are essential for informed 
decision-making. Measurement without comparison is not enough 
to judge the meaning of the results. (Contact: Wurster, Cecil, 
R., Division of Biometry and Applied Sciences, National 
Institute of Mental Health, 5600 Fishery Lane, Rockville, MD 
20857.) 

There is no shortage of information about cost analysis 
methodologies per se; however little has been written about the 
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application of these methodologies to psychosocial 
rehabilitation programs. Nor has auch been done to develop 
simple and inexpensive tools for assessing costs and financial 
stability in such programs. 

Approach * This form of cost evaluation addresses two 
questions: whether the agency has sufficient resources to meet 
its expenditure obligations, and more importantly, whether the 
agency has the needed resources and financial stability to 
provide the level and quality of service that they are charged 
to provide. Clearly a most important area of cost evaluation 
from the agency's viewpoint, the latter also is of import to 
funding agencies if they are to identify and contend with 
financially unstable agencies. 

Hall (1982) identifies four problem areas that can be 
addressed in evaluating an agency's financial condition, and 
lists 21 financial indicators that may be used to evaluate the 
extent of its financial problems. 
Indicators of a Declining Revenue Base 

1. Percent revenues per client served 

2 . Percent of one time revenues 

3. Percent of operating deficits 

4. Percent of elastic revenues 

5. Percent of expenditures for repair and maintenance of 
fixed assets 

6. Percent of unfunded pension liabilities 

7. Percent of unrestricted fund balance 

Indicators of Dependence on Unstable Revenue Sources 
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8. 


Percent 


of 


revenues from government 


9. 


Percent 


of 


revenues from memberships and contributions 


10. 


Percent 


of 


revenues from client fees 


11. 


Percent 


of 


restricted funds 


12. 


Percent 


of 


available sources of funds used 


13. 


Percent 


of 


unreimbursed overhead 
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Indicators of Increasing Costs for Services 

14. Personnel costs per client served 

15. Percent of expenditures for support personnel 

16. Percent of expenditures for fixed costs 

17. Fringe benefits liabilities 

18. Percent of nonlabor costs 

Indicators of Inadequacies in Fiscal Policy and Management 

19. Liquidity 

20. Percent of interest earned from investments of current 
assets 

21. Contract efficiency 

As Hall explains, it is not necessary to use all 21, nor are 
all indicators necessarily included. Young (1982) would add 
some indicators that gauge the adequacy of working capital, 
arguing persuasively that nonprofit as well as profit service 
agencies must have working capital in order to acquire fixed 
assets and to finance growth. Dritna (1980) suggests still 
others . 
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III. PROCESS EVALUATION 

The second set of evaluation activities that should be added 
to an overall evaluation strategy are those that assess the 
integrity, responsiveness / coherence and comprehensiveness of 
the program* Process evaluation focuses on the day to day 
operations of the program and is a way of understanding the 
extent to which service delivery is living up to the norms and 
ideals of psychosocial rehabilitation practice. 

Program Practices 

Background . With the exception of the standards developed 
by the Council on the Accreditation of Rehabilitation 
Facilities, there are no other national accreditation standards 
that can be used to assess the content of psychosocial 
rehabilitation programs. However, there are at least two other 
tools that can be used to measure services against the governing 
principles and definitions that are commonly associated with 
such services. An excellent exconple of such an approach is the 
Community Systems Workbook , prepared by DeSisto and Ridgway 
(1986) for the State of Maine. The document provides a means 
for systematically assessing the extent to which the program's 
mission is consistent with psychosocial rehabilitation 
principles and whether the key system components are in place. 
The second tool is the Psychiatric Rehabilitation Agency , 
Assessment process developed by the Center for Rehabilitation 
Research and Training in Mental Health at 'Boston University. 
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Much like accreditation in that it is paid for by the agency 
btiing assessed ^ it measures the protjrain against defined 
psychiatric rehabilitation principles and competencies. 

Approach . Two questions should be addressed: is an 
adequate service mix present, given the agency's mission; and 
are the services which are in place structured to reflect good 
psychosocial rehabilitation practices? Finally, the critical 
elements in the environment periodically must be operationalized 
and assessed, since the heal'ih of any program is dependent on 
the nature of the environment that is created for staff as well 
as for clients. (This final set of concerns is dealt with in 
Output Evaluation in the section on Staff Productivity.) 

As noted above the Community Systems Workbook spells out 
systeiu principles and articulates the components of a 
comp-u'ehensive program that meet the needs of persons with 
chronic mental illness. The service system functions include 
integrating services, basic supports, treatment services, and 
rehabilitation services. Each section is broken down into its 
component parts and specific discussion questions are posed. 

For use with staff and board members, the Workbook is an 
outstanding tool to determine periodically the vitality of the 
progra!>.. The Workbook can be augmented with questions developed 
by staff, clients and members of the board. Exhibit 6 displays 
the discussion cjuestions under, "Linkage, Coordination, Advocacy 
Functions" (integrating servicec) . 

The Psychiatric Rehabilitat on Agency Assess' snt Program, at 
the Boston University Center for Rehabilitation Research and 
Training in Mental Health, also provides a mechanism for 
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EXHIBIT 6 

Facilitatinsj Linkasjes^ Coordination^ Advocacy 

Comprehensive plans require linking psychiatrically disabled persons 
to multiple services, supports and resources. Without active 
coordination, many psychiatrically disabled persons will not follow 
through on referrals, agencies may not folloK through on initial 
plans, and the clients may be denied access to services and riupports 
"they require. Advocacy may be necessary to ensure access. Continued 
coordination ensures that all providers and resources are working 
together rather than at cross-purposes. 

GOAL - TO FACILITATE THE LINKAGES BETWEEN THE BASIC SUPPORT, TREATMENT, 
AND REHABILITATION SERVICES THAT EACH PSYCHIATRICALLY DISABLL7 PERSON 
NEEDS. 

LINKAGE, COORDINATION^ ADVOCACY FUNCTIONS 

1. How are patient/clients linked to services? 

Does this include information and referral? 

familiarizing client with community and resources? 

rehabilitation referrals which specify patient/client and providers 
roles and desired outcome? ^ 

individual advocacy for access to services and resources? 

other? 

2. Are ther active on-going linkages (e.g., joint planning, routine contact 
joint service delivery) among agencies in the area including: 

mental health agencies? 

housing authority? 

private landlords? 

health agencies ^ ^ 

social service agencies . 

community organisations/groups 

transportation 

educational programs 
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vocational programs 



psychosocial programs _ 
' recreational resources 

crisis services 

legal services 

other 



Is the agency/service area significantly involved vith interagency 
coordination: 

at the individual client level? . 



at the local systems level? 



4. Is the. J an agency or staff member that assumes a leadership role in 
facilitating coordination? 



5. Are there cooperative agreements: 
on an interagency level? 



6. Is there ongoing intensive vork among providers to ensure continuity 
and coordination? 



Source: Desisto, M.J. & Ridgeway, P., Community Systems Workbook , 1986, 
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assessing both the coherence of the agency's program and its 
consistency with best practices in psychiatric rehabilitation. 
A fee is charged by the Center to conduct the evaluation. 

Implementation and Analysis . We recomi\iend that program 
administrators approach program practice evaluation from 
multiple perspectives. It .o is suggested strongly that board 
members , staff and clients be involved in the assessments 
described. Finally, it is suggested that such reviews occur at 
least once per year. 

Program Connectedness 

B ackground . In evaluating psychosocial rehabilitation 
services, the importance of system variables and system impacts 
has been widely recognized (Carling, 1984; Tessler and Goldman, 
1982) . System level analysis is particularly important in this 
sphere of mental health because only through cuch assessments 
can .he complexity of the system components be captured, the 
"brokering" role of psychosocial rehabilitation b'3 illuminated, 
and the agency's relationship to the comjnun-'.ty that it serves 
be explored. 

In spite of its significance there has been little empirical 
work conducted in this area, partly because of the conceptual 
and methodological problems in pursuing system level research. 
For instv\nce, service systems are difficult to define because 
boundaries are not necessarily clear cut. Furthermore, service 
systems may have vertical (hierarchical) relationships; 
therefore, the level or levels must be delineated before 
measurement can begin. 
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Bachrach has pointed out that indigenous conununity support 
systems, or "organic" systems, are particularly difficult to 
evaluate or make generalizations about because of the 
idiosyncratic nature of their structures and operations 
(Bachrach, 1982). CSP systems, are based on ten system 
components and specified formal linkages, have greater potential 
for systematic study. But even these "synthesized" service 
systems have tended to emphasize discrete programs rather than 
service linkages. The result has been a lack of system level 
data CO analyze. Thus, causal relationships between system 
level variables and impacts jn clients and on the larger system 
remain elusive. 

Despite the conceptual difficulties and attendant 
methodological problems in measuring system variables and their 
impacts, several researchers have developed conceptual 
appro.^ches for analyzing systems and several have applied them 
to CSP initiatives or other psychosocial rehabilitation models. 
Morrissey, Tausig &. Lindsey vlS85) has analyzed a hypothetical 
community support system, using his paradigm for exploring 
organizational linkages. The basic procedure is to define 
system boundaries, to collect data ire :u key informants at as 
many levels of the organization as possible, and to use 
sociometric techniques or statistical clustering to analyze the 
data.' The units of analysis are pairs of organizations. 

Fein and Applegate (1982) conducted an empirical analysis 
of community support systems at the local level. For each CSP 
component, they identified provider agencies and categorized the 




quality of the relationships between each pair of providers. 

Another conceptual approach to system level measurement has 
been '^continuity of care*" Using the client as the unit of 
analysis, rerearchers have traced clients* movements through the 
system over a period of time (Bass and Winrtle, 1973; Tessler, 
Willis, and Gubman, 1986), thus making discontinuities 
apparent. However, it must be noted that not all 
discontinuities can bs attributed to gaps in the system or to 
poor communication between services . Clients themselves 
sometimes may elect not to follow thrvough on recommended 
services. The concept of continuity of care, as measured by 
client utilization of recommended services, is only partially 
useful in obtaining a measure of system linkages. 

Approach . Two key ingredients in psychosocial 
rehabilitation programs are programs* interconnectedness with 
other human services and the brokering function that they play 
on behalf of their clients. In their approach to the provision 
of services these programs tend to be multi faceted as opposed 
to unidimensional. The first issue to consider is the extent to 
which an agency has developed adequate connections with generic 
housing, income maintenance, employment, and other relevant 
generic an/ specialized human services . As persons with chronic 
mental illness have multiple needs, the quality of interagency 
linkages is crucial to successful client outcomes. Some of the 
methods for assessing the presence and efficacy of interagency 
relationships, which were described above, are extremely complex 
and not necessarily geared to an exploration of one agency's 



network. A more direct way to determine whether such linkages 
are in place is to assess client outcomes and zhe extern: zo 
which agency clients have been successful in securing needed 
generic services. The techniques described below involve both 
key informant interviews and self /assessment mechanisms. 

Instrrtmentation . Agency staff or members of the agency's 
board of directors can conduct key informant interviews with the 
relevant generic agencies in the agency's community as one 
approach to determining the intensity and utility of interagency 
linkages. Interviews should be based on a structured inquiry 
guide that is focused generally on the quality and consistency 
of interagency connections. Agencies that might make up the 
program's extended network of contacts include the following: 

• coirnunity mental health center; 

• state hospital or county general hospital; 

• private psychiatric hospital; 

• medical facility; 

• private therapists ; 

• public health or visiting nurse; 

• local welfare department; 

• local housing authority; 

• vocational rehabilitation; 
« local school system; 

• Social Security Disability Determination Service; 

• alcohol or drug abuse program; 
-i clergy/church groups; 

• legal services ; 

• police and law enforcement ; 
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• developmental disabilities and/or mental retardation 
agency; 

• transitional and group residential services; 

• sheltered workshop; 

• mental health advocacy groups; 

• private volunteer organizations ; and 

• other community agencies . 

Understanding the nature of interagency relationships 
requires a set of categories. The following taxonomy was 
developed by Fein and Applegate to review the nature of 
interagency relationships among community support services and 
other agencies in Iowa (1982) . 

1. Case management (CSS case managers consistently are 
involved in securing, coordinating or facilitating a 
particular service with the provider agency, on behalf 
of the client) ; 

2. Collaboration (CSS staff consistently are involved in 
joint staffingsr planning meetings, consultation with 
provider agency with regard to clients or CSS program) ; 

3. Referral (CSS stafj. refer client to provider agency or 
provide informatirn on agency person or resources); 

4. No linkage exisLs* 

Prior to conducting intervi'-ws with agencies in the network, 
staff should determine the nature of the relationship between 
relevant agencies as they see it. Agencies in the network 
should be asked to characterize their relationship with t]r^. 
program using the same categories. This should make it possible 
to validate the staff perceptions regarding the nature of their 
relationships with other community agencies. Additional 
questions ca:i ue asked, and responses can be elicited using a 
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five point scale (livery good, 2=good, 3=inedicore, 4=fair, 
5«unsatisf actory) : 

Please rate the nature of the information provided by the 
psychosocial rehabilitation program to facilitate 
referrals/collaboration, etc, ; 

Please rate the level of knowledge of your program among 
staff you deal with from the psychosocial rehabilitation 
program; 

Please rate the frequency of your contacts with the 
psychosocial rehabilitation program; 

Please rate the appropriateness of the referrals made to 
your agency by the psychosocial rehabilitation program; 

Please describe the ways in which existing relationships can 
be improved; 

Additional questions that are based on the particular 
circujiistances surrounding the network of relationships also can 
be developed. The important thing is to cast the net fairly 
broadly and to evaluate the connections on a regular basis. 

Another means of testing the viability of interagency 
linkages is through a self assessment process that proposes a 
series of problem scenarios in/olving various interagency 
activities. In a questionnaire developed as part of a two part 
exploration of the interagency relationships between mental 
health and vocational rehabilitation agencies and the connection 
between such relationships and positive vocational training 
outcomes, Woy and Dellar * d exemplified this problem solving 
technique. It was designed to be administered by a third party, 
but the problem statements also can serve as a self assessment 
exercise. The ratings after each problem, instead of coming 
from the interviewer, can be derived from a group consensus. 
Exhibit 7 reproduces the Interorganization Conditions portion of 
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the Woy and Dellario instrument. Although this particular tool 
is aimed at vocational rehabilitation and mental health 
agei:cies, the problem statements can be tailored to fit other 
agen.cy relationships • 

Match to Need 

In a perfect service system one could expect complete 
congruence between services needed and those received. In the 
real world, however, this occurs only sometimes. Numerous 
factors undermine the agency's ability to provide services 
needed by specific clients — eligibility criteria thwart case 
managers' efforts to secure needed services, some services such 
as housing are chronically undersupplied, many clients require 
services that cut across the boundaries of numerous agencies, 
and sometimes clients refuse to participate in services that 
case managers recommend. 

The goal of improving the degree of congruencr*^ between 
service needs and services received remains a reasonable one. 
Reducing the gap between services needed and services received 
is a key indicant of system responsiveness and maturity. 

Instrumentation and Approach , The service record 
described in an earlier section (Exhibit 2) illustrates one way 
of collecting information to assess the degree to which clients 
are receiving needed services. Case managers record the number 
of units of each service received over a specified time interval 
(column two on this service record) . Data from this column can 
then be compared with the service need information described 



EXHIBIT 7 
INTERAGENCY INQUIRY GUIDE 



Agency 

Respondent 
Rater 



Interorqranizational Conditions 
A^ Communications 

Problem: 1. The (VR/KH) requires information about one of your 
clients. Describe the procedure by which you share 
client information with (VR/MH). 



Issue: How well do the agencies share client information? 
Rating: 

very well well fair poorly very poorly 
1 2 3 4 5 



Interorcanizational Conditions 
A. Communications 

Problem: 2. The (VR/HH) wishes to make a client referral to your 
agency. Describe the procedure by which the client 
referral is made. 



Issue: How well do the agencies perform with respect to client 
referral? Rating: 

very well well fair poorly very poorly 
1 2 3 4 5 



Interorcranizational Conditions 
A. Communications 

Problem: 3. The (VR/KH) finds that they are able to offer a service 
complimentary to yours. Describe the procedure by which 
that service is coordinated with yours. 

Issue: How veil do the agencies coordinate services? Rating: 

very well well fair poorly very poorly 
1 2 3 4 5 
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Interorganizational Conditions 



Decision-malcing 

Problem: 4. You and your counterpart at (VR/MH) must decide on 
vhat services for a particular client should be 
provided by each agency. Describe the procedure by 
Khich this decision is made. 



Issue: Hov well do the agencies delegate responsibility for 
service delivery? Rating: 

very veil veil fair poorly very poorly 

3 2 3 4 5 



I. Interoraanizational Conditions 
B. Decision-making 

Problem: 5. You and the (VR/MH) are preparing a joint conference 
for working with a particular; client. Describe your 
input into this process. 



Issue: Hov veil do the agencies coordinate activities? Rating: 

very veil veil fair poorly very poorly 
1 2 3 4 5 

I. Interoraanizational Conditions 

B. Decision-making 

Problem: 6. The (VR/ME) is unable to live up to one of the 
preconditons of your linkage. Describe the 
procedures by vhich meeting these preconditions are 
monitored and corrected. 



Issue: Hov veil do the agencies conduct f ence-mending/peacsmaking 

activities? Rating: 

very veil veil fair poorly very poorly 
1 2 3 4 5 
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I. jCn terorqanizational Conditions 
C* Resources 

Problem: ?• You find that the (VR/MH) is experiencing a financial 

crisis and requires a change in case planning. Describe 
the procedures by which your agency responds to thi* 
situation. 



Issue: Hov well so these agencies exhibit flexibility of response to 
each other's problems? Rating: 

very well well fair poorly very poorly 
1 2 3 4 5 



II. Intraorganizational Conditions 
A. Communications 

Problem: 8. Some information about a new iutra-agency service to 
clients must be transmitted to members of your staff. 
Describe the procedure by which this information is 
transmitted. 



Issue: How well does this agency demonstrate intra-organizational 
communication? Rating: 

very well well fair poorly very poorly 
1 2 3 4 5 



II. Intraorganizational Conditions 
A. Commmunications 

Problem: 9. A potential client wishes to learn about the services 

provided by your agency. Describe the procedures by which 
such information is communicated. 

Issue: How well does this agency conduct outreach and public relations 
concerning its services? Rating: 

very well well fair poorly very poorly 
1 2 3 4 5 



Source: Voy, J.R. & Dellaroi, D.J., Issues in the Linkage and 

Interqrration of Treatment and Rehabilitation Services for 
Chronically Mentally 111 Persons . un::ated. 
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earlier. Two levels of analysis are possible here. At the most 
basic level, clicrits can be scored according to whether they 
received any amount of each needed service. A more detailed 
level of analysis would involve a comparison of the amount of 
each service needed and received.. Differences in either 
direction (i.e., less service or more service) would be worth 
noting. if clients are routinely receiving fewer units of 
service than case managers prescribe, this might suggest that 
the program's capacity needs to be increased or interagency 
linkages improved. A less frequent outcome is when clients are 
receiving more units of service than prescribed. This might 
suggest that the program is fostering dependency or, 
alternatively, that a less appropriate service is being used 
because the more appropriate service is unavailable. 
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IV. OUTPUT EVALUATION 
Output evaluation should be ths third area added to a 
comprehensive strategy for assessing psychosocial rehabilitation 
programs. Results regarding this dimension will assist program 
administrators to assess the health of their agency through a 
variety of efficiency measures. 

Productivity Evaluation 

This form of cost-related evaluation addresses the question 
of whether the level of effort or service produced, relative to 
the amount of resources expended (costs) is reasonable. 
Productivity is measured using a unit cost index that includes a 
measure of cost and a measure of output, m the case of 
psychosocial rehabilitation services, the costs that would be 
indicated are those of the program overall or by individual 
service. Productivity evaluations almost always are done 
comparatively in order to provide decision makers with a 
sufficient number of referents for judging the agency's relative 
productivity. For this reason, it is important that the 
measures of both outputs and costs are such that they can be 
applied consistently from agency to agency. 

Ir. any evaluation of agency productivity, there should be 
three components: (1) the identification of agency and/or 
service specific costs, (:>) the analysis of unit costs, and 3) 
the analysis of unit costs as a function of organizational and 
client variables. 

Cost evaluation involves the assessment of the costs of 
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producing psychosocial rehabilitation services. Depending on 
the type of evaluation, costs nay be defined in two ways: 
narrowly, to include only those actual financial outlays 
associated with producing the psychosocial rehabilitation 
services, or more broadly, to represent that set of opportunity 
costs and social costs that is associated with the provision of 
these services. The following sections include a discussion of 
cost finding as well as cost function analysis. 

Cost Finding . 

• Approach 

A definition of the costs associated with psychosocial 
rehabilitation agencies or programs is prerequisite to an 
evaluation of them. Of primary concern are psychosocial 
rehabilitation agency accounting costs, which represent the 
financial outlays involved in producing psychosocial 
rehabilitation services. 

As does any enterprise, psychosocial rehabilitation agencie 
record expenditures by object {e.g., salaries, fringe benefits, 
supplies, etc.}, record revenues by source {e.g.. Title XIX, 
state general funds, etc), and prepare periodic income and 
expenditure statements and balance sheets. These agency level 
data generally are sufficient to compute the necessary 
indicators for evaluating an agency's financial condition as 
explained in a previous section. 

On the other hand, identifying the total costs by service 
within an agency in order to evaluate its relative productivity 
in producing particular services can be a more demanding task. 
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This is especially true if the agency does not have a management 
information system that organizes costs by service. 

Sorensen and Phipps (1975), and Sorenson and Newman (1985), 
provide well written guidelines on how to design management 
information systems that support thoroughgoing, service specific 
cost finding procedures. These references should be consulted 
by agencies interested in developing such systems, 

• Implementation and Analysis 

The following guidelines are intended for those evaluators 
of psychosocial rehabilitation programs and service costs who 
must conduct ad hoc cost finding efforts as part of an 
evaluation of an agency's productivity, cr of program 
cost/benefits or cost/effectiveness* Even where agencies 
possess such management information systems and cost finding 
procedures, the evaluator still may find it necessary to 
superimpose his or her own procedures in order to assure that 
the cost finding is uniform across agencies. 

The procedure is divided into nine steps, the first six of 
which are sufficient to identify the total agency level or 
program costs. Steps six through nine are necessary to estimate 
the total cost by service. 

First, list the total agency expenditures by object for the 
period of interest. List the categories (objects) of 
expenditure down the left column. Across the top list the 
services of interest within the agency. Use footnotes to 
identify costs estimated for contributed items, to indicate how 
facility and equipment expenditures were adjusted (if at all), 
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and to explain the bases (e.g., square feet of occupied space, 
hours of direct staff time) upon which the indirect costs were 
allocated across different services offered (direct cost 
centers). An illustrative worksheet is shown in Exhibit 8. 

The personnel segment of the list of expenditures should be 
broken down by position, and fringe benefits should be shown 
separately. Other categories of expenditure should be broken 
down as far as the evaluator finds necessary for the purposes of 
the cost evaluation . 

Second, identify and add in any other expenditures that do 
not appear on the agency's books because they are covered under 
the budgets of related agencies (e.g., utility costs, 
transportation costs, insurance costs, pension fund costs, 
etc. ) . 

Third, identify and deduct any items of expenditure for 
activities and services that are of no material benefit to 
psychosocial rehabilitation agency clients (e.g., general 
research activities the potential benefits of which extend 
largely to clients other than those in the agency, and training 
of interns most of whom will leave the agency following their 
period of intership) . 

Fourth, adjust the facility and equipment items of 
expenditure (optional) . Some agencies simply may lease 
facilities and equipment, other agencies may expense all such 
purchases in the year of purchase, others may amortize these 
facilities and equipment over their useful lives, others (some 
public agencies) may not expense them at all. Unless these 



6& 



-63- 



EXHIBIT 8 



ILLUSTRATIVE WORKSHEET FOR ESTmTING COSTS 



ITEH OF 
EXPENDITURE: 



ALL: 



Total Indirect Direct 



COST CErfTER: 



Service A 
Indirect Direct 



Service B 
Indirect Direct 



Personnel 

Salaries & 

Wages 
Fringe & 

Benefits 

Facilities & 
Equipment 

Unadjusted 
Adjusted 

Purchased Svcs 

Utilities 

Supplies & 
Materials 

Other 



Total 



a Contributed (use to label contributed items) 
b Adjusted (explain the basis for adjustment) 

c,d^...n Allocation Base (use to identify base used to allocate indirect co: 
e.g. salaries, square feet, direct costs, etc.) 
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reported expenditures are made reasonably uniform, they can skew 
comparative cost evaluations. If the object of the cost 
evaluation is to assess actual, financial outlays^ then 
comparability is not an issue and such an adjustment would not 
be necessary*, A fairly common method for uniformly expensing 
facilities and equipment is to identify what it would cost to 
lease the facilities and items of equipment at current market 
rates, and to substitute the lease rates for the actual costs. 

Fifth, identify and show separately the market value of 
contributed goods and volunteer services that are not reflected 
rn the books. Counting these costs will result in cost f i^ ires 
that exceed actual (out of pocket) costs. Ignoring these costs 
will result in cost f-^gures under valuing the resources employed 
in providing psychosocial rehabilitation services. In either 
case, these costs should be identified and their treatment 
explained in any evaluation of agency costs. 

Sixth, sum the total program costs. 

Seventh, for each xtem of expense, estimate the portion of 
the expense that is direct and indirect. The direct portion 
represents that portion of the costs identified specifically 
with a particular service. The indirect portion is that which 
is incurred to the joint benefit of a number of services and 
thus must be apportioned among them. 

Eighth, assign the direct costs and allocate the indirect 
costs among the services. The indirect costs should be 
allocated on the basis of some common denominator {e,g., number 
of clients, direct costs, square feet of space) that fairly 



represents the utilization of the resources covered in the 
indirect cost figures* For example, a reasonable basis for 
allocating the facility maintenance and repair costs among 
services would be the square feet cf space occupied by each 
service area. An important principle of cost estimation is to 
devote attention to any particular category of expense according 
to the proportion of the total expenditures reflected by that 
component. Because psychosocial rehabilitation agencies are 
labor intensive, with personnel generally representing from 65% 
to 85% of the total costs, a procedure for tracing staff time 
over the period of interest to each of the services is central 
to any cost finding procedure. Unfortunately, the most accurate 
procedures for assigning personnel time and associated costs by 
program or service are also the most intrusive and expensive — 
independent monitoring and the institution of a staff time 
keeping system. Simply asking staff to break down how they 
spend their time in retrospect is a poor substitute, and can 
lead to unreliable results (Ashbaugh and Allard, 1934). 

Unit cost figures are simply found by dividing the total 
agency- or service-specific costs for a period identified by the 
number of units of output provided during that period. Three 
types of units (output measures) might be employed: (1) cost 
per case or episode, (2) cost per time interval (day, month, 
year), and (3) cost per period of service. For long term 
services, from which the client is not expected to graduate and 
of which the volume per client is not considered to be a prime 
indicator of the efficiency of the program, the cost per time 
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interval is probably the most appropriate measure. However, the 
cost per period of service may be the most appropriate measure 
for short term services, for which the volume per client is not 
considered to be a prime indicator of service efficiency. For 
other programs, in which the volume of services per client 
and/or the time spent in the program are considered to be prime 
indicators of efficiency, cost per case or episode is probably 
the most apt measure. For most psychosocial rehabilitation 
agencies, some combination of these measures is probably 
appropriate . 

Cost Function Analysis « 

• Background s 

The analysis of the variation in unit costs relative to 
variations in the programmatic, organizational and client 
factors (which combine to represent the program or service) 
commonly is termed "cost function analysis" or "production 
function analysis." At the base of cost function evaluation 
lies the concept that a given service can be produced 
alternatively by more than one set of resources (inputs). 
Moreover, the same service can be produced by a variety oZ 
different techniques for any set of resources. In other words, 
resources and techniques are substitutes in producing any 
combination of services. 

The purpose of such analyses is to identify those factors 
that explain the differences found in unit costs as part of the 
comparative analysis (e.g., program size, ownership 
(public/private), staff to client ratios, number of years in 
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operation, average staff tenure, etc J, Cost function analyses 
are an essential part of any comparative cost evaluation. After 
all, cost comparisons are useful only to the extent that one is 
able to understand what they represent* To point up differences 
in psychosocial rehabilitation costs among agencies is of little 
help to decision makers unless they also understand why these 
costs vary, whether the reasons seem justifiable, and whether 
they can do anything about it. 
• Approach 

If there are a large number of agencies subject to 
evaluation, multiple regression analysis can be used to identify 
client, programmatic and organizational variables (independent 
variables) that show a high (significant) degree of association 
with unit costs. One might employ a simultaneous regression 
procedure that forces the inclusion of all independent variables 
(found in earlier research to be predictors of unit cost) into 
the regression equation. Or, one might employ a stepwise 
regression procedure that orders the inclusion of independent 
variables according to their part correlation with the unit cost 
variables. In the latter, only those variables that 
significantly add to the predictive power of the linear 
regression equation become part of the final equation. 
• Methodolocrical Considerations 

Irrespective of the number of agencies under evaluation, the 
evaluator should identify those agencies that show the highest 
and lowest unit costs in the comparative cost analysis. Through 
interviews with key programmatic and fiscal informants and 



through the comparative analysis of these variables, he or she 
also should attempt to identify those variables that seem to 
account for the relatively high or low unit costs. The choice 
of extremes allows one more readily to uncover explanatory 
variables than would be the case if one were to select providers 
randomly. Because psychosocial rehabilitation programs are 
labor intensive, the first variables to be explored should be 
those relating to staff salaries, fringe benefits and staff mix 
(type of staff). Other factors that tend to differentiate high 
and low cost programs are differences in the amount of 
c :*ntributed resources and the type of clients served (e.g., 
acute versus chronic, old versus young) . 

Staff Productivity 

Background. The productivity of an agency's staff is 
inextricably linked to the way in which the agency is 
structured, the procedures that are employed, and the level of 
support that is provided. in the monograph. Assessing and 
Improvi ng the Performan of Psychosocial Re- hbili tation Staff , 
Friday and McPheeters (1985) note that there are several 
strategies available to psychosocial rehabilitation programs to 
improve productivity: 

• clarify program philosophies and goals; 

• improve organizational structures and procedures; 

• improve utilization of human resources; 

• modify clinical technologies; 

• improve organizational climate; 

o improve scheduling and use of time; and 
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• improve use of facilities and support services. 

Awroaeh . Before implementing these strategies, however, 
psychosocial rehabilitation administrators first must evaluate 
their programs in each of these areas. Exhibit S, taken from 
the 'Friday "and McPheeters report^ provides some saiaple questions 
in each of .he above areas. 



-70- 



EXHIBIT 9 
STAFF PRODUCTIVITY 



• Procrram Philosophies and Goals 

— It shall be the policy of the program to serve as many clients 
as possible vith a basically adequate quality of service, rather 
than only a fev clients vith the highest quality of service. 

— It shalll be the policy of the program to use those clinical 
technologies which are least consuming of time and staff 
resources in obtaining results for clients. 

--•It shall be the policy of the program to perform only the 
diagnostic and rehabilitation procedures vhich are substantially 
and significantly to individual client needs (not "routine" or 
tangential procedures and services) . 

It shall be the philosophy of the program to focus on improving 
the social and vocational functioning of clients rather than on 
their interpersonal functioning. 

— It shall be the philosophy of the program to particularly 
address those client needs and behaviors which, if not addressed, 
will very likely result in the client being rehospitalized. 

Orcraiiizational Structures and Procedures 

—Do the organization's structure and procedure create delays in 
transferring cli3nts or clinical information between services or 
units? 

Are there delays in implementing action because of requirements 
for decision and approval by too many individuals and team or 
staff meetings? 

Are there "standard operating procedures" to vhich all clients 
are subjected vhether they are needed or not? 

— Are responsibilities and authority sufficiently clear so that 
staff members or team leaders can respond readily to client needs 
and take approriate actions? 
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Dtilization of Human Resources 

— Are clinical staff assigned to be responsible for the overall 
care of individual clients or only to discrete activities of their 
professions? 

~ Are teams organized and used so that decisions can be easily 
made? 

— Is considerable staff time spent in meetings (e.g., quality 
assurance comaitees, planning and program evaluation committees)? 
These areas require staff participation, but often committees can 
function more expeditously vith a smalll number of staff, thus 
freeing clinicians for client work. 

" Is supervision adequate to identify and remedy problems, such 
as excessive absenteeism, poor vork habits, or clinical 
deficiencies? 

—Are paraprof essionals used as client care workers or only as 
aides to professionals? 



Modified Clinical Technologies 

—Could behavioral approaches be used to shorten the 
rehabilitaion time and improve the results? 

—Can group approaches be used for some of the education and 
counseling of clients, and thus increase the numbers of clients 
served? 

~ Can short-form assessment procedures suffice for most clients 
instead of the traditonal full-scale instruments and procedures? 



Improved Orcranizational Climate 

—Is there organizational instability as a result of frequent 
reorganizations or leadership changes? 

Is morale poor as a result of low pay or bureaucratic 
rigidities? 

— Are there communication problems that leave staff feeling 
unsure or that nobody cares? 

Are there incentive and reward systems for outstanding 
performance? 
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Scheduling and Use of Time 

—Are there provisions for scheduling alternate client for times 
vhen clients are late or cancel their appointments? 

— Are staff conferences, training sessions, etc,, held to the 
minimum time necesssary and at times vhen they are least likely 
to interfere vith client needs? 

— Are vorking hours, lunch hours, etc,, scheduled so that there 
is maximum time available for staff to vorJc vith clients? 

— Are the vork and data scheduled so that they flov from one 
staff member to another vithout undue delays? 

— Do staff members have adequate skills in personal time 
management? 



Use of Facilities and Support Services 

— Are team members located so that they can interact easily? 

—Are there support staff to assist in record keeping, monitoring 
cases and appointments, and preparing reports? 

— Are automation systems used to streamline staff activity? 

—Are statistical reporting forms simplified for staff use, and 
is information fed back to them promptly? 



Friday, J.C. & McPheeters, H.I., Assessing and Improving the Performance 
of Psychosocial Rehabilitaion Staff . 1985. 
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V. OUTCOME EVALUATTOW 

Evaluating outcomes is at the crux of program evaluation. 
Continued social, political, and financial support depend 
ultimately on the program's ability to demonstrate that it has 
made a difference in the lives of the client's it serves. 

In this section a few of the many possible outcome 
dimensions are discussed. In thinking about outcome research, 
one tends to think first of service effectiveness . However, 

acceptability is also a relevant dimension. Therefore, 
. we have included in this section discussions of client and 
family satisfaction with services. 

The section ends with a review of client outcome research, 
focusing on community adjustment, level of functioning and 
quality of life. 

Client Satisfaction . 

Backaround . Given the primacy of client autonomy among 
psychosocial goals, it is somewhat surprising that client 
satisfaction studies are fairly rare in the literature on 
psychosocial programs. One exception is Hill House in 
Cleveland, Ohio, which has intentionaally set out to assess 

» 

client satisfaction and has also included clients themselves in 
the design of the evaluation process. Rill House has formed a 
special program — client Oriented Program Evaluation, or COPE 
— in which a group of volunteer clients design evaluation 
instruments, collect data, interpret results and implement 
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recommendations. Volunteers are paid a small stipend. To date, 
they have developed a needs assessment form and a client status 
instrument, and have assessed client satisfaction with specific 
Hill House services. 

Hill House has identified the following as advantages of 
client participation in the execution of client satisfaction 
surveys: 1) higher face validity of client designed instruments; 
2) greater integration of program evaluation into other agency 
activities; 3) greater openness on the part of respondents to 
client designed items; and 4) more serious consideration of 
findings by planners and practitioners (Tanaka, et al., 1984). 
The use of clients as evaluators ~ in addition to respondents 
— is discussed in the concluding chapter. 

Other examples of client satisfaction instruments which 
attempt to captur-^ the client's service experience as well as 
the impact that services might have had on his or her life 
include the "Member Satisfaction Questionnaire" prepared by 
Fellowship House (December 1986) and the "Client Satisfaction 
Questionnaire" (Attkisson and Pascoe, 1983). Both of these 
instruments include specific questions both about the 
performance of the agency as well as the influence that the 
service had on the client's return to social and vocational 
functioning. 

Approach. Client surveys have been used to solicit che 
opinions of service consumers in a variety of health and human 
services areas. Client surveys may be used alone or in 
conjunction with other monitoring or evaluation techniques as 
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part of a larger evaluation plan- Data may be collected through 
personal interviews, telephone surveys, or self-administered 
questionnaires. Such evaluation activities, however, are 
relatively recent, especially in mental health. Evaluation has 
been focused primarily on the professional and technical aspects 
of service. Studies have been conducted by professionals and 
only rarely has client input been considered. 

This situation has changed considerably during the past 
decade and client surveys are gaining increased credibility. 
This new interest in client attitudes reflects increased 
consumer activism (through the formation of political and self 
help groups) and an intensified demand for accountability from 
those who provide services- It is also a recognition that 
notions of program effectiveness should be multifaceted and 
defined by a range of interests including those of the 
re'cipients of service. 

In addition to assessing the level of satisfaction with 
services, client surveys also can elicit information relevant to 
staff and agency performance, quality of life „ d service 
outcomes- This is discussed in other sections of the 
monograph- Finally, the use of client satisfaction assessments 
provides a validation of the agency's mission; it is a way cf 
expressing to persons who use services that they are equal 
participants in the service process. 

Assumptions and Methodological Considerations . The design 
of any client survey should conform to the general principles of 
sound research design- A number of specific technical 
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considerations complicate this design. These include maximizing 
the reliability and validity of data obtained from clients, and 
establishing norms which make the data from client surveys more 
easily interpreted. 

• Reliability and Validity 

The literature on client satisfaction suggests that data 
from client surveys may be especially vulnerable to problems of 
reliability and validity. The former is defined as the 
consistency of a measure over repeated applications; the latter 
as the degree to which an item actually measures the concept it 
is intended to r.easurs. There are a number of sources of error 
in client surveys that can reduce considerably the reliability 
and validity of the measurement process. But there are 
strategies that appear to improve the reliability of data ~ 
specifically the use of multi-item rather than single item or 
global measures. 

The validity of measures used in client surveys is 
threatened by the tendency of respondents to try to give 
socially acceptable responses to questions. In their efforts to 
conform to widely accepted beliefs or attitudes, respondents 
sometimes distort their true sentiments. This problem affects 
all forms of survey research; however, it is especially 
troublesome in client surveys. Since clients frequently are 
dependent on the service provider in a variety of ways, they may 
be reluctant to express sentiments that are critical of the 
service provider. They may feel that negative comments could 
result in the withholding of services or a reduction of effort 
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exerted on their behalf. If a client feels constraints on his 
ability -xpress concerns about a service program or progress 
in the program, the validity of resulting data is suspect. 

Fortunately^ there are a number of strategies that make 
clients less reluctant to provide valid feedback. Perhaps the 
most obvious is to assure respondents of the confidentiality and 
ancnynity of their responses. If clients know that all data are 
to be aggregated and that no personally identifying information 
will be released, . they jnay feel more comfortable in expressing 
their true sentiments. These assurances typically are provided 
in an introductory statement at the beginnirj of a questionnaire 
or in the introductory comments made by an interviewer. 

In the case of client interviews, measurement validity also 
can be improved by having a person other than the client's 
primary ser^-ice provider conduct the interview. It is almost 
always bad practice to have an individual who is associated 
closely with the treatment or service process asking clients for 
feedback concerning the program. If the program has a research 
staff, these individuals might be appro riate interviewer:^ . If 
}iot. clerical staff can be recruited for the task. Another 
approach is that of training volunteers, board members or other 
clients to pose questions. 

Simlarly threatening to measurement validity is, 
-acquiescent response sef (ARS) , or the tendency of re.spondents 
to agree with statements on a questionnaire or interview form 
regardless of content. Ouce again, this is a problem for survey 
research generally, but it is probably more pronounced in client 
surveys. Strategies to minimize the effects of ARS center on 



the instrument design. For instance, where it is possible, 

———'- — -—-•a. ~ j<-a/nv< j.5j^wiiSc onwuj-ia us avoiaea. wnere 
this is not possible, check items should be inserted into the 
questi.onnaire or interview schedule to insure that acquiescing 
is not happening {e.g., paired questions, such as "are you 
usually happy?"; "are you usually sad?"). Also, it is wise to 
ask questions about the injtrumental character of the services 
received, such as, "in what ways has this service expanded your 
opportunities to participate in the activities of your 
community?" 

• Developing Norms 

The ability to compa^re scores for the same organization over 
time or to compare scores from simi ir agencies enhances most 
evaluation studies. This is especially true of studies that use 
client surveys to obtain feedback on satisfaction with 
services. Since client satisfaction surveys typically tend to 
shorf fairly high rates of satisfaction, these scores by 
themselves do not always provide a lot of information. By 
comparing these scores with available norms, however, it becomes 
somewhat easier to interpret small variations. In the case of 
psychosocial rehabilitation programs, the norms most likely will 
be derived from previous applications of the survey. But it may 
be possible to standardize client satisfaction surveys within a 
region or within a state, thus generating norms across pre grams. 

Specific In'>truments. Exhibit 10, "Client Evaluation of 
Services" (Larsen, Attkisson, Hargreaves, & Nguyen, 1979), is an 
example of a simple and straightforward instrument, easy to 



EXHIBIT 10 



CLIENT EVALUATION OF SERVICES 

Please help ur improve our program by answering some questions about the 
services you have received. Ve are interested in your honest opinion, whether 
they are positive or negative. Please answer all of the questions. Ve also 
welcome your comments and suggestions. Thank you very much, we really 
appreciate your help. 

CIRCLE YOUR ANSWER 

!• How would you rate the quality of service you have received? 

4 7 2 ^ 



Excellent Good Fair 

Did you get the kind of service you wanted? 



Poor 



No, definitely not No, not really Yes, generally Yes, definitely 
To what extent has our program met your needs? 



Almost all of my Most of my needs Only a few of None of my 

needs are met have been met my needs are met needs are met 

If a friend were in need of simlliar help, would you recommend our program 
to him or her? 

1 2 3 4 

No, definitely No, I don't Yes, I think so Yes, 

not think so definitely 

Hovr satisfied are you with the amount of help you have received? 



Quite Indifferent or Mostly Very 

dissatisfied mildly dissatisfied satisfied satisfied 

6. Have the services you received helped you to deal more effectively with 
your problem? 

4 3 2 1 

Yes helped Yes helped No really No made 

a great deal somewhat didn't help problems worse 

7. In an overall, general sense, how satisfied are you with the service you 
have received? 

^ 3 2 1 

^^^y . Mr.stly Indifferent or Quite 

satisfied satisfied mildly dissatisfied dissatisfied 

8. If you were to seek help again, would you come back to our program? 
1 2 3 4 

definitely not No I don't think so Yes I think so Yes definitely 

Source: Larsen, Attkisson, Hargreaves, & Nguyen, Service Evaluation 
Questionnaire . 1979. 
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administer and score. It has multiple items and it avoids 
yes/no responses. Depanding on the nature of the particular 
program, one may want to ask specific questions about service 
components such as residential and vocational supports. One 
also may want to ask more open ended questions about the ways in 
which the program contributed to personal autonomy and 
independence, and the extent to which staff displayed respect 
for individual dignity. 

Another instrument which may prove useful was developed by 
Fellowship House in Florida. Their client survey asks general 
question:* about barriers to service, membership and 
transportation issues, comfort with existing communication 
networks, discrimination, and the gr»neral effect that the 
program has had on the lives of the consumers. Specific 
questions also are asked about the social, residential and 
vocational components of the programs. Included in the appendix 
is the full 55 item quer ^ionnaire with the 1985 results of one 
set of norms for purposes of comparison. Though the Tellowship 
House questionnaire is long, it does include more material on 
program related satisfaction, such as the treatment clients 
received from staff and the extent to which the program has 
facilitated integration. 

Impleme ntation/Analvsis . Client satisfaction surveys 
should be conducted consistently and systematically on a 
schedule that depends on program need. As noted earlier, they 
can be administered in a variety of ways: using research or 
clerical staff, asking clients to provide responses in writing, 



using volunteers or board Tnenbers to ask questions, or asking 
other clients to help with the process. The latter technique 
probably would ensure a more frank and open response. The key 
is to assure clients that their responses will be confidential 
and that there will be no repurcussions to cri^'-.ical answers. 

Family Satisfaction 

Backgrounds The search of the literature revealed few 
evaluation studies in which families served as respondents. An 
excellent review of these articles can be found in a paper by 
Fisher, Benson and Tessler (1987). 

The studies that have been conducted suggest that families 
in the caregiver role are dissatisfied with the kinds of 
services that they are offered (Hatfield, Fierstein, and 
Johnson, 1982). Schulberg and Bromet (1981) emphasize the 
importance of assessing the satisfaction of families as part of 
an overall .evaluation scheme: "Alleviating family burden 
through an increased range of services in more hospitable 
agencies is thus a major goal of community support systems. 
Assessments are needed of whether this target population has 
benefited. " 

Among the studies which were found, three stood out. The 
first, ^y Hatfield, F.\erstein and Johnson (1982), was based on a 
survey of members of ;:^elf-help organizations of families of 
mentally ill persons in niixe states. Respondents were asked to 
provide demographic information, their goals for treatment, and 
the extent to which such treatment met thoso goals. The results 
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strongly indicated that the families' needs were not being 
served by their current service arrange^T.ents . The study 
findings were compared in turn with various theories of family 
therapy to determine whether family goals were consistent with 
the goals as espoused by the field- in commenting on the lack 
of consonance between family and professional goals, the authors 
note: "Families are not asking for help for their own problems 
but rather for the unusual levels of knowledge and skill they 
need to effectively manage mental illness at home. Mental 
health professionals understand the need for training those who 
care for the mentally ill in settings other than the home; yet 
families' difficulties in management are ascribed to pathology 
and they are offered therapy." 

Conducted by Levine and Spaniol (1985) , the second study 
also involved a survey of members of self help groups composed 
of family members of persons with mental illness. The purpose 
of the survey was to secure information on the extent of family 
involvement ua facilitating and/or providing services to persons 
with mental illness. The authors :ommented on the relative 
benefits and drawbacks of family sponsored community services. 

An additional study by McLean, Greer, Jcott and Beck 
(1982), provides an evaluation of a support and education group 
for families of person.*; with mental illness whose relatives had 
been institutionalised at a state hospital in Massachusetts. 
The article describes the level of staff involvement, the nature 
of the training provided to parents, the goals expressed by 
family members, and the ultimate establishment of a parent 
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self-help group outside of the hospital. All three articles, 
indicate a strong desire among family members for concrete 
information about their mentally ill family member's condition 
and the treatments and medications available. 

An article by Reiter and Plotkin (1985) describes a study in 
which family members assumed a far more active role. In this 
instance family members actually monitored conditions at a state 
hospital and monitoring reports were given to unit directors as 
feedback concerning needed improvements. 

Clearly, additional evaluation is necessary to probe further 
into the areas of family satisfaction, family capacity building, 
and family services most likely to assist families to cope with 
the needs of their mentally ill family member. As noted 
earlier, it is also important to understand how family 
expectations rliffer from those of the client and in turn to use 
this information to design programs that seek to find a common 
ground of understar iing. 

Approach. As our literature review suggested, relatively 
little is known about the interactions of families and 
providers. One can speculate that this relationship could be a 
stressful one, given the level of emotional involvement and the 
strains inherent in caring for persons with chronic and serious 
mental .illnesses. 

Families are playing an increasingly important role in the 
community care of persons with chronic mental illness. We need 
to know more about how relationships between families and 
providers evolve and how service systems can enhance these 
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relationships (Fishe>-, Benson, and Tessler, 1987). Some of this 
is basic research arxd should be conducted on a large scale. 
However, the individual program also has an interest in this 
issue since it bears directly on the welfare of clients and or 
successful program outcomes. 

Before a program can begin to involve family members in 
evaluation and/or monitoring, decisions concerning the level of 
involvement need to be made. Involvement could range from 
having family members respond to satisfaction questionnaires to 
the more active role of monitoring service delivery. In the 
following section we discuss one way of initiaing far.ily 
involvement in the evaulation process through a very brief 
family satisfaction questionnaire. 

Assumptions and M ethodolocical Consideration?:; . Most of 
the methodological considerations which need to be addressed in 
a research plan to evaluate family satisfaction have already 
been discussed in the section on client satisfaction. 

Evaluation strategies need to ensure that family respondents 
feel free to express honest opinions, without fear that critical 
comments will have negative repurcussions for their family 
members. Survey formats that promise anonymity can make 
respondents feel considerably freeer to express their concerns. 
Mail surveys that do not require names or other identifying 
information are a good exan.ple of such formats. 

The problem of normative comparisons is particularly acute 
with respect to family satisfaction studies. Since so little 
has been published in the research literature, it is difficult 
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for programs to compare their levels of satisfaction with those 
other similar programs. Also, since the satisfaction 
questionnaires should be completed anonymously, it is impossible 
to use respondents as their own controls, thus measuring changes 
in satisfaction levels over time. 

In this case, however, the absence of norms is not a serious 
flaw/ since the main purpose is to use the survey to identify 
incipient problems and stress while there still is time to 
launch remedial action. 

Instrument , There are few instruments to choose from in 
the area of family satisfaction. One interesting approach has 
been developed by Hatfield (1987). It has been used by the 
Alliance for the Mentally m chapter in Montgomery County to 
assess family satisfaction, A slightly modified version of this 
instrument is attached as Exhibit 11, 

The instrument differs from some of the more typical 
satisfaction instruments, which tend to look only at global 
assessments of service acceptability. It focuses on 
satisfaction with specific dimensions including: 
o Appropriateness of expectations; 
o Growth in independence from family; 
o Care and supervision; 
o Growth in living skills; 

o Assessment of relative • s satisfaction; and 

o Provider/family relationships. 
In addition to scaled questions, one might add some open 
ended questions which seem appropriate to the particular 
situation , 
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EXHIBIT 11 

FAKILY SATISFACTION VITH COmi^lITY SEVICES 

Ve are making a brief assessment of family satisfaction vith Comiaunity 
Services, If your relative nov uses these services or has done so in the past 
tvo years, please reply. . . 

Community Services 

^^t® satisfaction vith community services for your relative . 

LOV 

Appropriate level of expectation 12 3 4 

Grovth in independence from family 12 3 4 

Care and supervision of relative 12 3 4 

Grovth and socialization and living skills 12 3 4 



Your relative's satisfaction vith living 
situation 

Relationship of staff to family 



1 
1 



2 
2 



HIGH 
5 
5 
5 
5 

5 
5 



Please explain any lov ratings {ratings of 1 or 2) 



Sign if you vould like to discuss this questionnaire: 

Name : ^ 

Address ; 

Phone:^ 



Source: Hatfield, A., Alliance for the Kentallv 111 . 1987. 
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Implementation/ Analysis , We recommend that the family 
satisfaction data be collected through mail surveys* These 
surveys could be conducted annually; or if resourc- ere 
sufficient and circumstances warranted more frequent 
administrations, every six mouths. The data can be used to 
identify potential trouble spots as well as to obtain a picture 
of overall relationships between families and the provider. 

Client Outcomes 

Backcrround. This section contains a review of a sample of 
outcome evaluations drawn from the voluminous literature on 
outcome research that has emerged during the past two decades. 
Much of the work that has been done in this area is quite 
specialized and involves fairly sophisticated statistical 
analyses and modeling. it has advanced considerably our 
understanding of the effects of different kinds of treatment. 
However, it also has made the technology of outcome evaluation 
appear somewhat inaccessible to program directors who have 
limited resources to devote to research. 

The purpose of this section on outcome is to suggest that 
research to date — or at least some pieces of it — can be made 
relevant to the program director or manager who is interested in 
feedback on how his/her program is doing, as opposed to general 
information on how psychosocial programs operate. 

The studies reviewed here are just a sample of the available 
literature. Our intent is to illustrate the ways in which 
various indicators have been used to measure treatment 



outcomes. The outcome measures reviewed include 
rehcspitalizationr economic self-sufficiency, living 
arrangements, community adjustment or level of functioning, and 
quality of life^ This section also contains a discussion of the 
possible types of normative comparisons that help one to 
interpret the findings from outcome evaluations. 

Rehospitalization . Rehospitalization has been the most 
frequently used measure of success for community mental health 
programs. The use of rehospitalization as an outcome measure 
has been criticized lately for a variety of reasons. As Mosher 
and Keith (1980) have noted rehospitalization can be influenced 
by a number of system-related variables. In addition, 
hospitalization can mean a number of different things. For 
example, a hospitalization episode of a few days' duration 
primarily for medication review is very different from a 
hospitalization episode that lasts for several months. It 
clearly could be quite misleading to treat bocL episodes as 
similar. 

Despice these caveacs, rehospitalization is still a useful 
measure, particularly when it is used in conjunction with other 
outcome indicants. Relatively easy to measure, it also bears a 
relationship to the goals of most community based mental health 
programs . 

Rehospitalization has been measured three ways in the 
research literature.. In its simplest form, rehospitalization 
can be treated as a simple dichotomized variable; clients are 
either reshospitalized during the tudy period or they are not. 
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Lamb and Goertztl (1972), and Mulkern et al. (1986) reported 
simple rehospitalization ratei'» as one of several measures of 
success for community programs. 

A second and somewhat more sophisticated approach involves 
an examination of the number of hospitalizations or the length 
of stay of each hospitalization during a specified time period. 
This allows one to get a somewhat clearer understanding of what 
each client's rehospitalizations mean. For example, Shedletsky 
and Voineskos (1976) presented data on the number of 
rehospitalizations , This allowed them to examine the "revolving 
door" phenomenon. Similarly, Dickey, et al, (1981) reported the 
amount of time clients had spent hospitalized during a study 
period and Smith (1975) reported the correlate of days 
hospitalized, and casting his data as percent of time spent in 
the conmunitv (i.e. not hospitalized or in any other 
institutional setting during a three year study period) . 

The third and best approach to analyzing rehospitalization 
data is exemplified by Radinsky, Rein, and Blanas (197S) , who 
collected data on the number of rehospitalizations during the 
twelve month periods preceding and following enrollment in a 
community program. This allowed them to use the subjects as 
their own controls and to examine changes in the pattern of each 
individual's use of hospital care following treatment. 

Economic Self-Suf f iciencv . A second variable which 
frequently has been used as an outcome indicant is economic self 
sufficiency. Many chronically mentally ill persons have 
virtually non--existent work histories and few skills that would 
serve them well in the work place. Recognizing the therapeutic 
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value of work in and of itself, the meaning of v^ork a:- an 
integral component of adult role performance, and the centrality 
of work in American culture, many community programs have 
developed vocational training programs or supported work models 
in their service programs. The Coitu-aunity Support Program, for 
example, has included vocational services as one of its core 
service components. 

Economic self sufficiency has been conceptualized and 
measured with a number of different indicators. Like 
rehospitalization, the simplest measure is a dichotomized 
variable of working/not working (Solberg and Raschman 1980). 
Employment status is a useful variable because it is easy to 
measure and bears a clear relationship to program goals. The 
problem with this gross measure, however, is that competitive 
employment may be an unrealistic goal for many persons with 
chronic mental illness. Field and Yegge (1982) broadened this 
dimension, examining "productive activity" rather than simple 
employment status. This they defined as employment, homemaker 
duties, volunteer work, and formal employment training. 

Another indicant of economic self sufficiency is welfare 
status. Some researchers (e.g., Auerbach and Pattison, 1976; 
Radinsky, Rein, and Blanas, 1978) have used the proportion of 
clients coming off welfare as an indicant of program 
effectiveness. Given the goal of most community programs to 
increase clients' ability to provide for their own economic 
needs, this is a reasonable outcome measure. But, in using this 
measure, one needs to be aware that simply coming off" welfare 
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may not necessarily indicate improvement in the client's ability 
to provide for him/herself* Systjm level changes, such as the 
wave of eligibility redeterminations for SSI and SSDI in the 
early to mid 1980s, may influence the client's welfare status. 

Living Arrangement , The client's living situation is a 
third indicator that is often used to evaluate the effectiveness 
of community programs. Most studies examining this topic have 
focused on hospital vs. community residence rather than the type 
of living situation in the community. However, the latter 
setting is a relevant outcome indicator, as most programs make 
implicit value judgments concerning the relative desirability of 
different types of sectings. Such judgements should be enhanced 
by information from client satisfaction surveys regarding 
residential preferences' and should be sensitive to individual 
circumstances . 

One way of utilizing this variable is to describe the 
living situation of program clients at one point in time. Any 
number of schemes may be developed to categorize different 
settings . 

A more detailed approach is to examine changes in living 
situation during a specified study period (Mulkern, et al., 
1986). This allows one to assess the stability of clients' 
residential situations over time. Such an assessment, however, 
should be tempered by an understanding of the extent of movement 
among the general populacion and among particular age groups. 

Level o f Functioning/Community Adjustmen t. One of the 
most heavily researched outcome areas is client adjustment or 
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functioning. The language used in various studies differs: 

some researchers refer to level of functioning, others to 

community adjustment, still others to psychosocial functioning. 

Most, however, are concerned with the client's ability to 

perform adult roles in the community and on the level of active 

symptomatology. This is obviously a wide area to explore. The 

diversity of methods used reflects not only the complexity of 

tne concepts being measured, but also the lack of agreement on 

the best measurement techniques. Some of the issues that remain 

to be resolved include; 

o Identification of the domains that are most 
useful in understanding community adjustment; 

o Agreement on measures that capture these do- 
mains efficiently, reliably and validly; 

o How best to measure change (whether to use raw 
change scores or some standardized change 
score that corrects for the individual's Time- 
I score) ; and 

o Identifying and controlling extraneous 

variables which might mediate the relationship 
between treatment and outcome. 

Despite these rather serious meaiiurement problems, numerous 

studies have attempted to measure client functioning or 

adjustment. Most of these outcome studies have at lease two 

things in common. The first is the use of a longitudinal 

approach to measurement; assessing functioning or adjustment at 

more than one point in time. Most er.ploy a simple pre-test/ 

post-test design with two data points, one at program admission 

and a second at some later point. The second common element is 

the use of mult-^'ple measures of functioning or adjustment rather 

than a single measure. This enables the evaluator to deal more 

ER?C 9 8 



*; -93- 

adequately with the complexity of the concept being measured and 
to explore the relationships among outcome dimensions 
therr.selves . 

A full review of studies which have examined level of 
functioning or community adjustment is beyond the scope of this 
section- However, a description of a sample of such studies 
will provide a sense of the diversity of approaches. One of the 
earlier studies of the impact of psychosocial rehabilitation 
services on level of functioning was conducted by Wolkon, Karmen 
and Tanaka at Hill House (1971). Employing a modified control 
group experimental design, this study followed the progress of 
clients released to Hill House from three state hospitals. 
Three dimensions of role performance were assessed: 1) social 
participation, 2) instrumental role performance, and 3) 
affective or expressive ro2 e performance. Social participation 
included auch activities as church attendance, 
inf ormalaorganizational involvement, and visits with friends, 
instrumental role aerformance included such family tasks as 
shopping and performing household chores. Affective role 
performance included the frequency with which specific topics 
were discussed by the client and his or her family and friends. 

Stein, Test and Marx (1975) examined adjustment as part of 
their evaluation of the Training in Community Living (TCL) 
program. The dimensions they included were living situation, 
time in institutions, employment, social relationships, quality 
of environment, subjective satisfaction with life, end leisure 
time activities. 
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Ross, Menapace, and Teitelman (1981) examined psychosocial 
functioning in their evaluation of clients enrolled in the 
psychosocial program at Horizon House. They used an instrument 
entitled "Client Adjustment Rating Scale (CARS)." This 
instrument includes scales measuring nine domains: motivation, 
self-concept, family functioning, self-reliance, affect and 
mood, vocational/educational readiness, interpersonal 
relationships, personal maintenance, and community resources. 

In a recently completed longitudinal study of clients 
enrolled in Community Support Programs, Mulkern, et al. (1986) 
examined changes in community adjustment using eight scales: 
acting out in the community, acting out at home, socializing 
with friends, community participation, community living skills, 
overall adjustment, grooming skills, and alcohol and drug 
problems . 

The above examples are a cross section of studies that use a 
variety of multi dimensional measures to assess outcome. They 
are instructive because they attempt to deal with the complexity 
of outcome measurement. Many studies augment these 
multidimensional scales with global assessment measures. 
Economical to administer and analyze, such measures often 
provide good summary indications of level of functioning or 
changes in level of functioning. As Endicott et al. (1976) have 
noted: 

The most commonly used global rating scales are simple five 
or seven point scales with single word adjective anchor 
points without any cues or criteria to aid the rater. It is 
a testament to the power of global rating scales that even 
such simple scales have been demons\:rat ed to be among the 
most powerful in detecting change. They cannot, or course, 
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supply the detailed inf or.Tnation provided by multidimensional 
rating scales, but they have their obvious uses as summary 
statements . 

One such scale is the Resource Associated Functional Level 
Scale (RAFLS) developed by Lef f (1985) . This is a seven-point 
ordinal rating 2=<:ale that links social role functioning and 
behavior to level and type of services needed. This scale 
differs somewhat from other scales described in the literature; 
it includes an assessment of both service need and level of 
functioning. Additionally, most global scales described in the 
literature were developed for use with a wide range of client 
types with drastically different: impairment levels. As a 
result, when such scales are used to assess persons with chronic 
mental illness, cases tend to be clustered at the bottom of the 
scale and the discriminating power of the instrument suffers. 
The RAFLS was developed specifically for persons with chronic 
mental illness and tends to spread cases somewhat more widely. 

This brief review of outcome studies examining community 
functioning or adjustment suggests that there is a fair amount 
of overlap in the areas covered by different evaluators. 'r^here 
also are unique aspects to most of the studies. The variety of 
measures used provides rich coverage of this complex phenomenon; 
the studies cited provide good examples of how various measures 
can be integrated into a comprehensive outcome assessment. 

Quality of Life . A final outcome dimension is quality of 
life. Unlike more traditional mental health programs that focus 
almost exclusively on symptom reduction and enhanced psychiatric 
status, psychosocial rehabilitation programs have broadened 
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their focus, taking the improvement of clients' living 
circumstances as legitimate targets for intervention. Quality 
of life is being recognized as an increasingly important 
dimension that needs to be addressed in any comprehensive 
assessment of outcome* 

A review of quality of life studies suggests a split of 
prof es:sional opinion as to whether an individual's quality of 
life is best captured by objective or subjective measures. 
"Objective" indicators might include such variables as an 
individual's income and housing situation. "Subjective" 
measures, called "psychological indicators" (Zautra and 
Goodhardt, 1979), focus on an individual's perceptions of his or 
her life. 

There is a variety of available quality of life instruments 
which conceptualize quality of life in a number of different 
ways. Some focus on self -reported happiness and perceptions of 
well-being (see, for example, Bradburn, 1969; Bradburn, 1965). 
Others focus on life satisfaction (Neugarten, Havighurst, and 
Tobin, 1961; Campbell, Converse, and Rodgers , 1976; Andrews and 
Withey, 1976; Flanagan, 1978); mental well-being (Beiser, 1974); 
and positive mental health (Jahoda, 1958; Wright, 1971; Zautra 
and Goodhart, 1979). There are also a number of instruments 
that focus on the more objective indicators of life quality 
(see, for example, Caro, 1981). 

Two QOL instruments have been used in a number of settings 
and will provide soLie indication of the types of approaches 
used. The first is the Oregon Quali?:y of Life Questionnaire 
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(OQLQ) developed by Douglas Bigelow, et al.; the second is the 
Quality of Circumstances of Disabled Persons, developed by 
Francis Caro, 

Composed of four sets of scales — personal adjustment, 
interpersonal adjustment, adjustment to productivity, and civic 
adjustment the OQLQ is administered to clients by trained 
interviewers . 

Caro's Quality of Circumstances instrument contains items 
that measure somewhat more objective elen.ents of the client's 
environment, though this scale also is sensitive to the 
individual's feelings concerning the adequacy of his life 
cir'^umstances. The instrument contains items relevant to nine 
domains: shelter, nutrition, sleeping patterns, personal care, 
clothing, activity and self expression, health maintenance, 
mobility, and choice. 

These instruments are fairly representative of those being 
used in QOL studies generally. While they provide a detailed 
and rich assessment: of clients' living circumstances, they also 
are expensive to administer and require a high level of 
expertise for che data analysis. 

Approach. Under the best of circumstances ^ *ie measurement 
of treatment outcomes is still a difficult task. Even if one 
manages to develop outcome measures that are reliable, valid and 
sensitive enough to detect change, the problem of causality 
remains. Given the multiple influences on an individual's life, 
it is often difficult to attribute to the program in question 
observed changes in behavior, functioning or any other outcome 
indicator. 
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Further complicating outcome measurement is the fact that 
projram goals often are articulated poorly and, in many cases, 
treatment goals are client specific (i.e., staff start with a 
client's status or level of functioning and identify a set of 
treatment goals that are appropriate for that individual). 
Outcome evaluation, as a result, needs to be sensitive to the 
very personal or individual nature of "success" in such 
programs, and methods appropriate to this orientation must be 
identified. 

The time frame in which such analyses must take place also 
complicates outcome evaluation in psychosocial rehabilitation 
programs. These programs typically serve populations that are 
characterized by mental disabilities that are severe and of long 
duration. As a result, client change can be expected to occur 
slowly, and frequently in a ncn-linear fashion. Outcome 
measures need to be sensitive to the chronic and episodic nature 
of these disabilities and to the relatively long time required 
before change is likely to occur. 

All of these factors result, in part, from the unique 
philosophy of psychosocial rehabilitation centers. They make 
outcome evaluation a complex task. In order to provide useful 
feedback on program performance, evaluation plans need to be 
sensitive to each of these issues. 

The remainder of this section presents a framework for 
conducting outcome evaluations which are feasible at the program 
level. Assumptions underlying the framework are identified, 
examples of instruments are presented and described, and 
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implementation and analysis issues are discussed. 

Assumptions and Methodological Considerations . Several 
assumptions underlie the recommendations which follow. The 
first is that outcome is a multi-dimensional concept; therefore, 
attempts to measure it also should be multi-dimensional. There 
is no single indicator that adequately captures the phenomenon. 
Our evaluation framework reflects this complexity and includes 
several outcome measures. 

The second assumption, somewhat self-evident, is that the 
outcome measures selected should be compatible with program 
philosophy. The measures recommended here are oriented toward 
level of functioning, community integration, and quality of 
life — all of which have a particular salience for psychosocial 
rehabilitation programs. 

The third assumption concerns the feasibility of conducting 
high quality, reliable evaluations at the program level. The 
outcome literature cited in the previous section provides 
numerous examples of very detailed and comprehensive evaluation 
studies, which cost a lot of money, involve some fairly 
sophisticated statistics, and were conducted by experienced 
researchers. For small programs lacking specialized research 
staff and other resources for research, some of these endeavors 
may appear to be a bit dciunting. However, we believe small 
programs can conduct targeted outcome evaluations which will 
provide useful feedback and can be used to monitor program 
operations and plan for program change. 

Our fourth and final assumption is that, given the 
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philosophy of psychosocial progran\s , outcome information should 
come from a number of different sources — not just the client's 
therapist. Sources should include the case manager/therapist/ 
staff, the client, and the client's family members or 
significant others. 

Instruments , Instrument construction is a time consuming 
task, generally involving extensive reviews of the literature, 
form design, pretesting, and redesigning on the basis of pretest 
information. In some cases, designing new forms is an 
unavoidable part of th.^i research process. But, for small 
programs trying to do credible evaluations amid the overwhelming 
demands of service delivery, it makes more sense to try to 
identify instruments, or sections of instruments, that already 
have been field tested and that are known to be reasonably valid 
and reliable. Exhibits 11 and 12 contain sections of 
instruments that measure outcomes and quality of life and that 
have been used with persons with chronic mental illness. 
• Outcome Measures 

The instrument attached as Exhibit 12 contains selected 
measures from the Uniform Client Data Instrument (UCDI) . The 
UCDI was designed by CSP stafi from/New York, Colorado, and 
Michigan, in collaboration uith staff from NIMK. The instrument 
subsequently was modified by Human Services Research Institute. 
The UCDI captures four areas of client related information: 
demographics, clinical history, adjustment to the environment, 
and service utilization . The measures included here c.re those 
that relate to outcome: employment status, living arrangement, 
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EXHIBir 12 



1* ladicare cliesc's currsn: living 
arraageaent (Circle one code only) 
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Inpaciest of a psychiaCvic 
hospital or facility 


01 


Skilled nursing facility — 
24 hours 


02 


Inter:3ediate care facility- 
less than 24 hours 


03 


Supervised group living 


04 


Transitional group hcne 


05 


Fanily foster care 


06 


Cooperative £pt., supervised 


07 


Cooperative apt, unsupervised 08 


'Board and care hcce 


C9 


Boarding house (including 
meals, no supervision cr 
progran) 


10 


Roocing or boarding house or 
hotel 


11 


Private house or apt. 


12 


No regular residence 


13 


Other (Specify) 


57 


Don't kr.ow 


98 



2* Does client have faaily living 
nearby (vithin one hour of 
driving tiae)? 



Yes and faaily is involved 
with client or staff 1 



Yes, but faaily is not 
involved with client or staff 2 



Yes, but don't knov if faaily 
is involved vith client or 3 
staff 



No 



Don't knov 



3a, Do^.s client current 1 v have any 
type of job? 



Yes 



No 



Don ' t kncv 3 
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3b* Indicate type of job. (Circle 5^* Indicate all kr.ovr. sources of 

one code only) client's iacoae (Circle all that 

• n n 1 V ^ 



Ccapetitive job (paid, 
obtained on the open siarket) 


• 

1 


Earned income 


01 


Transitional eoployaent (paid 




Social Security Beaefi:,: 


..02 


Job through CS?) 


2 










Social Security Disability 




Work tr;*ining (pre-vocational 




Income 


03 


readiness, unpaid) 


3 










Supplenental Security Inccne 


04 


Sheltered workshop 












Araed Service connected 




Volunteer position, Specify: 


5 


disability payrients 


05 


Other type of job. Specify: 


6 


Social Welfare Benefits-state 
or county (ADF,food staaps) 


06 



Don't know 8 



Vocational progran (CITA, 
vocational rehabilitation, 07 
sheltered workshop) 



Unecploynent cocpensation 08 



If client is not currently 






employed, indicate why. (Circle 
one code only) 


Retirement, investment or 
savings , pension 


09 


Temporarily laid off * 1 


Rent supplements 


10 


Physically disabled/unable 

to work 2 


Alimony anc child support 


* il 



Mentally disabled/unable 

to work 3 



Retired A 

Don't know . 98 



Student (full-time) 


5 




Hcmemaker ( full-time ) 


6 


5b. Provide your best estimate of the 
client's total mcnrhlv income 


Other (Specify) 


7 


(before taxes) zrom all sources* 



Q Don't know 
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Jamily/spouse contribution 12 



Other sources SPECIFY 97 



. Gross Monthlv Ir.cosje 

1 '.y 



6. Indicate Che client's performance level on each of these Comaunity Living 
Skills and Socialization Skills during the past month s (Circle one code only 
for each skill.) 





NO 
PROBLEM 


MILD 
PROBLEM 


MODERATE 
PROBLEM 


SERIOUS 
PROBLEM 


DOES NOT DON'T 
APPLY KNOV 


Uses available 
transportation on 
familiar routes 


. 1 


2 


. 3 


4 


6 8 


performs household 
chores 


1 


2 


. 3 


4 


6 8 


Prepares/obtains 
own r.eals 


1 


2 


3 


4 


6 8 


Maintains adequate 
diet 


1 


2 


3 


4 


6 8 


Uses iwailable 
transportaion oa 
uafaoiliar routes 


1 


2 


3 


4 


6 8 


Manages available 
funds 


1 


2 


3 


4 


6 8 


Secures necessary 
support services 


1 


2 


3 


4 


6 8 


Engaged in recreational 
activities at home 


1 


2 


3 


4 


6 8 


Socialized with friends 


1 


2 


3 


4 


6 8 


Socialized with members 
of family 


1 


2 


3 


4 


i 8 


Engaged in scheduled 
daytime activates 


1 


2 


3 


4 


6 8 
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Question 6 NO MILD MODlRATE SERIOUS DOES NOT DON'T 

continued , . . PROBLEM PROBLEM PROBLEM PROBLEM APPLY KNOW 



Attended club lodge or 



other meetings 1 2 


3 


4 


6 


8 


"Attended church or 

other religious svcs 1 2 


3 


4 


6 


8 


Engaged alone in 

recreationAl activities 1 2 
out side home 


3 


4 


6 


8 


Engaged with others in 

recreational activites 1 2 

out s ide home 


3 


4 




8 


7. Answer the following questions about Cr 
code* (Circle one code for each Question) 


iminal Act 


ivites 


using the 


frequency 




NOT AT 
ALL 


ONE 
TIKE 


2 OR MORr 
TtMES 


DON'T 
KKOW 


Has the client been picked up or arrested 
for any type of crime in the last nonrn? 


1 


2 


3 


8 


Has the client been picked up or arrested 
for any type of crime in the last year? 


1 


2 


3 


8 


Has the client been a victim of criminal 
activity in the past month? 


1 


2 


3 


8 


Has the ciiwut been a victim of criminal 
activity in the past year? 


1 


2 


3 . 


8 



110 
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8. Indicate client's perforaance level on each of chese Personal Care Skills & 
Inappropriate Behaviors during the past month , (Circle one code for each iten) 





NO 
PROBLEM 


MILD 
PROBLEM 


PROBLEM 


SERIOUS 
PROBLEM 


DOK*T 
KNOW 


Incontinence 




2 


3 


U 


8 


Walks/gets around 




2 


3 


A 


8 


Dresses self 




2 


3 


U 


8 


Maintains personal hygeine 




2 


3 


4 


8 


Made suicidal threat or attempt 




1 


3 


4 


8 


Had trouble at work cr school 




2 


3 


4 


8 


Had trouble in household 




2 


3 


4 


8 


Destroyed/ stol e property 




2 


3 


4 


8 


Had trouble with the lav 




2 


3 


4 


8 


Made violent threats or attempts 




2 


3 


4 


8 


Caused comnunity coaplaints 




2 


3 


4 


8 


Exhibited temper tantrums 




2 


3 


4 


8 


Engaged in bizarre behavior 




2 


3 


4 


8 


Caused complaints from household 




2 


3 


4 


8 


Abused drugs 




2 


3 


4 


8 


Abused alcohol 




2 


3 


4 


8 
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- level of incoiae, source of income/ family involvement, community 
living skills, socialization skills, personal care skills, and 

' criminal activities and/or victimization. 

These particular measures were selected because they reflect 
community functioning and integration, the two outcome areas 
that are most consistent with the philosophy of psychosocial 
rehabilitation. The UCDI has been field tested in two large 
survey studies (Market Facts, 1981; Mulkern, et al., 1986). 
Case managers, therapists, or other staff familiar with the 
client are the sources of information. 
• Quality of Life Measures 

Exhibit 13 provides an example of a quality of life 
instrument that is appropriate for use with clients with chronic 
mencal illness. Measures on this instrument come from four 
sources: the Oregon Quality of Life Questionnaire (Bigelow, et 
al. , 1980), the Quality of Circumstances of Disabled Persons 
Survey (Caro, undated), the Client's Quality of Life Interview 
(HSRI, 1984), and the Social Network Inventory (Anderson, 1979). 

Measures included on this instrument are: leisure time 
activities, adequacy of finances, satisfaction with residence, 
satisfaction with neighborhood, and satisfaction with 
relationships . 

Implementation /Analysis . 

# Longitudinal Data 

In order to measure changes in level of functioning (as 
opposed to level of functioning at a particular time) data have 
to be collected at multiple time points. Time 1 and time 2 
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EXHIBIT 13 



QUALITY OF LIFE 



Ve are interested in hov you are doing in the Community 
Support Program. First, I vould like to ask you about 
how you spend your time. 



On a usual day, how much time do you spend (a/b/c) : Is it all. of your 
time, most of your time, or none of your time? 







ALL 


MOST 


SOME 


NOKE 


a. 


with friends? 


4 


3 


2 


1 


b. 


with relatives? 


4 


3 


2 


1 


c. 


alone? 


4 


3 


2 


1 



How often do you find yourself sitting around with nothing to do? 
Is it? 



all of the time, 


4 


most of the time, 


3 


once in a while, or 


2 


never? 


1 



Vbat vould you like to do with your time? (PROBE) 



Don't know 8 



Vhat stops you from doing (this/these things)? (PROBE) 
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5. Do you have a regular place to live? 



yes 


1 


no 


2 



6. Do you have enough money each month to cover: 





YZS 


NO 


food? 


1 


2 


clothing? 


1 


2 


rent? 


1 


2 


medical care? 


1 


2 


traveling for work, chopping, medical appts? 


1 


2 


traveling to visist friends and relatives? 


1 




social activities like movies or restaurants? 


1 


2 



7. I vill read a list of things about your home. Please tell me if you 
are very satisfied, satisfied, dissatisfied or very dissatisfied with each 
of these things: (REPEAT RESPONSE ALTLRNATIVLS VEEN NECESSARY) 





VERY 
SATISFIED 


SATISFIED 


DISSATISFIED 


VERY 
r:SSATISFIED 


its condition or 
state of repair 


1 


2 


3 


4 


the amount of room 
or space it has 


1 


2 


3 


4 


the amount of 
privacy you have 


1 


2 


3 


4 


its security or 
safety 


1 


2 


3 


4 


your neighbors 


1 


2 


3 


4 


its convenience to 
stores St shopping 


1 


2 


3 


4 
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8. Would you say that the neighborhood where you live is: 



very safe, 


4 

X 


safe if you are careful 


2 


a little dangerous, or 


3 


not safe? 


4 



9. Overall/ has your relationship with the people at (name of CSP) made 
your life: 



much better 


1 


a little better 


2 


no different 


3 


a little worse, or 


4 


much worse? 


5 



END OF INTERYIEV 
Thank you for helping me by answering these questions. 

Source: Uniform Client Data Instrument (NIMH), Human Services Research 
Institute, 1978. 
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scores then can be compared to give an indication of the 
presmned effect of treatment. 

However, even with longitudinal data, change scores from a 
single program are difficult to interpret by themselves. ".n the 
absence of comparison data, it is nearly impossible to establish 
whether a program is successful. For example, suppose that an 
evaluation study revealed that one-quarter of clients discharged 
to the community from psychiatric hospitals were readmitted to a 
hospital during a twelve month follow-up period. By itself, 
this figure is difficult to interpret. If other programs 
experience higher recidivism rates, then this 25% percent figure 
would 30und good. On the other hand, if recidivism rates of 10% 
or 15% were the norm, then this program would be hard pressed to 
consider itself successful. The literature on outcome research 
in psychosocial rehabilitation programs and other similar 
community based programs contains examples of a number of 
different ways of handling this need for comparisons or norms. 
The experimental design is the classic and most rigorous method 
employing random assignment to treatment and control groups and 
pre- and post-test measures. 

For a numbei of reasons, one of which is ethical 
considerations, the classical experiment is rarely feasible in 
ongoing programs. A more practical approach, and the one 
recommended here^ is to use research subjects (clients) as their 
own controls. This involves a comparison of the client's status 
on outcome indicators before and after treatment (or at two 
different points in time during treatment) . 
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# Data Collection 

Data for the outcoine questionnaire are supplied by the case 
manager, therapist, or other staff member familiar with the 
client. The information that is needed to fill out the form can 
be obtained from client records or on the basis of the 
respondent's knowledge of the client. In order to reduce b\as, 
data for the quality of life questionnaire should come directly 
from the client. These data could be collected through 
interviews or by a mail survey. 

In both cases, the first data collection point constitutes 
the baseline. These data can be used by themselves to describe 
the caseload in terms of functioning, employment, and quality of 
life. Second and succeeding administrations will collect data 
that can be compared with baseline figures to document progress 
over the study period. Data should be collected at least 
annually, but if resources permit, a six month interval could be 
used. 

• Data Analysis 

The first task of this type of analysis is to examine the 
distribution of change scores over time. Simple tables can be 
constructed to show the percent of clients who improved during 
the interval, the percent who remained stable, and the percent 
who deteriorated. This is an interesting exercise in and of 
itself; however, the data are somewhat difficult to interpret. 
For example, how much collective improvement does one need to 
observe before you are willing to call a program a success? One 
advantage of using instruments which have been fielded elsewhere 
is that therf. are norms available against which a program can 
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measure its performance. 

An equally interesting question is who changes? Most 
studies of treatment outcome use a fairly standard approach. 
The first step is to use correlation analysis to get a sense of 
which variables are related to the dependent variables and which 
independent variables are related to each other. ' After 
examining the zero order correlations / multiple regression can 
be used to suggest the relative importance of each variable and 
the proportion of rhe variance that is explained by the 
particular set of independent variables. 
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VI. OUTPUT /OUTCOME 



Background . 

Very little of the literature reviewed for this project 
.included systematic cost effectiveness assessments. Those 
authors that addressed the issue usually used very simple 
calculations of per diem rates derived from a division of the 
number of clients into total agency budgets (Cohen, Sichel, and 
Berger, 1977; Mulder, 1982; Alaska, 1984; Stein, Test, and 
Marx, 1975) . A few of the materials reviewed did describe more 
sophisticated analyses of effectiveness and cost indicators 
(Smith, 1975; Weisbrod, Test & Stein 1980; Cogswell et al., 
1985), but these were the exceptions. 

One project worth noting for future reference is that being 
undertaken under contract to the National Institute of Mental 
Health (Contact: Paul Widen, Division of Biometry and Applied 
Sciences, National Institute of Mental Health, 5600 Fishers 
Lane, Rockville, MD, 20857) . A cost-outcome methodology for 
application with programs serving personr» with chronic mental 
illness has been developed and is being field tested in two 
states. The Uniform Client Data Instrument (UCDI) is being used 
to collect the client outcome indices. The project is to be 
completed on December 31, 1987. 

We also reviewed surveys of cost effectiveness studies. 
Rich, Bednarz, Westra, and Goldsmith (1984) describe several 
case examples that illustrate the application of cost benefit 
methods, providing step-by-step descriptions of the completa 
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process with flow charts and diagrams. Rich and Goldsmith 
{1982} critique several approaches to cost benefit analysis that 
focus on cost measurements. They note the importance of 
discounting the time value of costs, identifying non-dollar 
costs r recognizing opportrnxty costs, and distinguishing between 
marginal and fixed costs. Frank (1981) discusses the problems 
of estimating non-quantifiable benefits of mental health 
services and briefly describes marginal benefit ana cost 
effectiveness analyses. He also includes several case studies 
of cost benefit analyses. 

Approach . 

The comparative evaluation of costs per unit of service 
allows one to provide reasonable assessments of the cost of 
providing services. But the ultimate cost evaluation measures 
are those that provide some indication of both the impact of the 
services on clients and of their value to society relative to 
costs. VThen the outcomes are valued in monetary terms the 
evaluation of costs relative to outcomes is termed cost/benefit 
evaluation; when the outcomes are valued in non-monetary terms, 
the result is cost effectiveness evaluation. Strictly speaking, 
outcome measures alone represent "true" measures of program 
efficiency on effectiveness. It is only with these measures 
chat one can tell whether the intermediate outputs have been at 
all useful — that they have had some desired effect. In these 
analyses, costs and outcomes tend to be defined broadly: they 
are intended to neasure not only client benefits and agency 
costs, but societal benefits and costs as well. 
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It is important to remember that cost/benefit and 
cost/effectiveness evaluations are intended to assist public 
policy makers in making choices among different programmatic 
approaches. They are not intended, nor are they appropriate, to 
inform choices among particular providers. Thus, these 
techniques are of import to psychosocial agencies only to the 
extent that the latter are themselves party to such evaluations 
and are in a position to influence them* Psychosocial 
rehabilitation agencies should not be subject to such 
evaluations individually. 

Cost/Beppfit Analvses 

Cost benefit analysis allows one to compare the costs and 
benefits to society of the various program alternatives for 
persons with chronic mental illness. As explained by Rothenberg 
(1975), "in its most refined form, the technique can be used to 
take account of such complexities as characterized by 
differences in the time allocation of benefits and costs and 
differences 5n who receives the benefits and who pays the 
costs...? Moreover, since there is an attempt to compare the 
monetary value of benefits with the monetary value of costs, the 
cost benefit calculus enables the evaluator to use a common 
yardstick to assess the relative attractiveness of 
alternatives. Thus, by calculating the costs and benefits of 
policy alternatives in terms of monetary values, one can compare 
such dimensions as rates of return on investment, net 
differences between costs and benefits (net present values) , and 
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be„efit-tc-=ost ratios. Of bourse no alternative would ba 
undertaken whose costs exce.d benefits, and in general the ones 

that would be selected would be tho». ■ ■ , 

oe tnose that naximized the total 

societal benefits relative to costs. - 

cost/benefit evaluations are important if policy .akers are 
to appreciate fully the larger costs and benefits to society 
that attend the psychosocial rehabilitation of persons w.th 
Chronic .ental ill„ess; however, the difficulty co.es in 
-signing monetary value to these costs and benefits. Resting 
on a series of broad assumptions, the bases or rationales for 
arriving at social cost and benefit estimates necessarily beco.e 

highly theoretical. In t-hi «: ^: 

in this type of cost evaluation, the costs 

involved in providing psychosocial rehabilitation services are 
defined as representing „ore then the financial outlays required 

to produce the services t>,«>.» 

rvices. There are opportunity costs and 

societal costs as well. 

opportunity costs include the costs of goods or services 

that are foregone by virtue of their expenditure for the 
psychosocial rehabiliation progra. or component service (s). 
Theoretically, the ti.e that a person with chronic .ental 
illness Who is capable of working (producing goods and services, 

spends in psychosocial rehabii ita^-; 

renaoiiitauion service represents a cost 

to the individual (i.e., lost earnings,, and a cost to society 
(i-, foregone production of goods and services,. Similarly, 
the .arket value of contributed goods and volunteer ti.e spent 
.n producing services would be counted as opportunity cost. Not 
only could they be employed for alternative purposes, but the 
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earnings that might accrue from the non-expensed (undepreciated) 
portion of the investments in program facilities and equipment 
would represent an opportunity cost, in that these funds could 
be invested elsewhere. 

Social costs could include a multiplicity of costs connected 
with the support of a person with chronic mental illness: the 
costs of law enforcement services for client.^ who are vagrant or 
who commit criminal acts.^ and the cash or in-kind contributions 
of the chronically mentally ill person's family. 

Benefits to society likewise may be viewed principally in 
economic terms using such indicators as discounted future 
earnings over a person's lifetime may be used, or less 
productivity oriented measures, i.**. as the economic value 
persons would place on their own lives using life insurance as 
the surrogate, or the maximum prices that members of society 
would be willing to pay to have the program carried out, A very 
comprehensive and clear explanation of alternative schemes for 
valuing benefits and costs is provided by Thompson (1080) . 

Cost Effectiveness Analyses . The assumption that the 
benefits or outcomes can be valued by their market prices or 
those of similar alternatives is crucial for performing 
cost/benefit analyses of alternatives. Yet, many of the 
outcomes of psychosocial reh; : litation programs have no market 
counterpart. For instance, what is the market price to attach 
to the improved social functioning of a person with chronic 
mental illness? What is the market price to attach to the 
improved ability of a person with chronic meatal illness zo live 
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independently in the community? Allowing one to express such 
outcomes in non-monetary terms, cost-effectiveness analysis also 
enables one to examine the relative costs of alternative 
programs for achieving particular types of outcomes. But it 
does not allow one to compare program costs directly with 
benefits. ' That is, the cost effectiveness approach enables one 
to rank potential program choices according to the magnitude of 
their effects relative to their costs / but not to ascertain 
Vfhether a particular program is "worth it" in the sense that 
benefits exceed costs. This is because the latter generally are 
expressed in monetary units; the former, rendered in 
non-monetary units of effectiveness. 
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VII. CONCLUSION 



This final section of the monograph includes a discussion of 
some general issues that administrators of psychosocial 
rehabilitation programs and others are likely to encounter as 
they implement an evaluation design; Topics briefly covered 
include the recommended phases of an evalution design, the types 
of individuals and organizations that can assist the 
administrator in carrying out an evaluation, the level of 
accuracy and prf^cision that should be reflected in the 
evaluation methcdology, and the use to which evaluation findings 
can be put. 

Prior to developing a plan for program evaluation, it is 
important to pause and review the reasons why evaluation is 
important. Insofar as the field of pyschosocial rehabilitation, 
evaluation is an important means of documenting program content, 
service approaches, and program results. Such documentation is 
important to the replication and expansion of psychosocial 
rehabilitation techniques around the country. Evaluation 
results also serve an important function with respect to funding 
bodies at the local, state and federal level. Specifically, 
results are important to document what those in the field 
already believe — that psychosocial rehabilitation programs are 
cost-effective alternatives to more restrictive, 
medically-oriented services . 
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As noted earlier, evaluation is important to the internal as 
well as the external functioning of the agency. Periodic 
evaluation activities provide the information necessary for 
managers to assess issues such as staff deployment, program 
practices, and program results. Evaluation provides a means for 
re-energizing the agency's program and providing staff with 
direct feedback about program efficacy and coherence • 

Implementing an Evaluation Plan 

The ultimate aim of evaluators within a psychosocial 
rehabilitation program should be to develop a comprehensive plan 
for assessing program viability and efficacy. Obviously, given 
resource limitations and the stage of development of an agency, 
such plans can not be implemented all at one time. Therefore, 
when choices must be made about which aspects of an evaluation 
plan to begin first, the initial focus should be on the capacity 
of the agency to provide psychosocial rehabilitation services. 
Capacity has been defined in this monograph to include the 
agency's mission, staff competency, and financial condition. 
Other areas that can be included in this phase include an 
exp3,oration of the appropriateness of the physical plant in 
which the program is located (e.g., is it human scale? 
comfortable? homelike, etc.), the location of the program (e.g., 
is it close to public transportation, is it in a safe area? 
etc. ) r and the representativeness of the board of directors 
(e.g, is it reflective of the community? does it include 
consumers as well as professionals? etcj 
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Once a mechanism is in place to assess capacity, the next 
phase is to craft a system for monitoring the conduct of the 
program and the extent to whicl* the service process embodies the 
principles of psychosocial rehabilitation. The monograph covers 
three important areas of the process of service delivery — 
program practices , program connectedness , and "match-to-need, " 
or the extent to which tiie sei'vices actually provided conform to 
what the client needs. Ot^ar s^r^^as of service conduct that can 
be taken into account during this phase include the quality of 
the staff and client interaction (e.g., are clients treated with 
dignity? are their vi^:iws and opinions solicited? etc.), the 
oversight of . medication administration (e.g., are regular checks 
made to determine possible side-effects? are clients informed 
about potential risks? is there a full medical history in the 
client's record? etc.). 

Following the creation of an evaluation scheme to ^issess 
service process, the next step is to set in motion a means for 
measuring program outputs. Those suggested in the monograph 
include ♦•he agency's financial as well as staff productivity. 
Additional outputs might include numbers of staff receiving 
P-.xodic skills training, number of board meetings held, numbers 
of clients dropping out of the program, and number of grievances 
filed. 

The next step in an evaluation plan is to assess the extent 
to which the services offered were successful from the 
point-of -view of the client as well as the client's family or 
other significant friends. Further, the monograph recommends 
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that program results also be assessed in terms of such client 
variables as securing employment, maintaining stable housing, 
and minimizing contact with the mental health system. Other 
outcomes not noted have to do with measured increases in staff 
competency, staff satisfaction, and increased community 
acceptance of persons with serious mental health problems. 

Finally, administrators can, during the final phases of an 
evaluation, link program outputs (costs per unit of service) 
vith client outcome information to generate a cost benefit or 
cost effectiveness ratio. 



Who Does Evaluation ? 

in large agencies, there may be a full-time staff member 
responsible for evaluation. However, in small psychosocial 
rehabilitation agencies, such as social clubs, imple^nenting an 
evaluation plan will fall on individuals with other staff 
responsibilities. it is useful, therefore, to think of 
evaluation as an agency and community wide endeavor that 
involves a variety of actors both as respondents and assesors. 
specifically, the evaluation design should contemplate 
participation by clients, families and friends, members of the 
board of directors, staff, and lay persons in the community. 
Additional information and references concerning the use of such 
groups can be found in Assessing and Krhancina the o,,...-^. 
Human Servires- A Guide fn^ the ^ d (Bradley, et al., 1985), 

g - ^^^^^" Evpluarion in P^..^.•... > - asebook on r^.^... 
Evaluation of Mental Health and nth.. ........ iBr.aiey, Allard 

and Mulkern, 1984) . 
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With respect to clients and families, they can function as 
respondents to surveys on satisfaction and client outcome, and 
can also be pressed into service to assist with evaluation 
design and to conduct key informant interviews. Members of the 
board of directors are a valuable resource to the agency and can 
serve a.^ observers of the service delivery process (e.g., to 
determine how clients are dealt with by staff, how client views 
are solicited, how the intake process is carried out, etc.), and 
can also survey community reactions to the program through key 
informant interviews. By giving board members a specific 
responsibility, it will enhance their connection to the agency 
and should expand their commitment. 

Individuals from the community can also be recruited to 
conduct an external evaluation of the prc^gram including 
interviewing staff to determine their perceptions of agency 
strengths and weaknesses as well as the clients and their 
families. Using persons outside the agency as a periodic 
evaluation resource enhances the agencies connections with the 
community and helps to build a basis of support for the 
program. Finally, staff at all levels can serve as respondents 
to surveys regarding process as well as outcomes, and can also 
function as interviewers of community key informants. 

Even in those agencies that do have staff assigned to 
evaluation positions, including a range of interested groups in 
the conduct of evaluation is important both because it is in 
keeping with the mission of psychosocial rehabilitation programs 
and it enhances the role and connection of each of these 
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n;l>^oJ;e are also technical resources that may be made 
available to the program for little or no cost. The first place 
to turn is to the local community or four year college. Faculty 
'in computer sciences may be willing to assist in setting up a 
simple set of file for evaluation results on a microcompter . 
Students may be interested in providing assistance as part of 
independent stud^ placements or for purposes of graduate theses. 
Members of the board of directors may have specialized skills 
such as accounting or public relations that may be useful in the 
evaluation design phases. 

Further, local and state Community Support Program staff may 
have written materials, reports, and guides in their libraries. 
They may also be willing to spend time with agency staff and 
assist in developing the evalulation plan. Finally, the agency 
may be in a position to seek pro bono assistance from major for 
profit management consulting firms. Many of the so-call ^'^^7 
eight" consulting firms have pro bono policies that allow their 
employees to contribute a certain amount of time to charitable 
purposes. This is also the case with some of the major computer 
companies such as IBM. 

Finally, it may be necessary to hire a consultant to assist 
in the development of the plan. In order *:o identify an 
appropriate individual, the administrator may want to contacc 
the International Association of Psychosocial Rehabilitation 
Services or the Non--Profit Management Association. 
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Evaluation Research vs. Program Evaluation 

Throughout this document we have tried to make the 
distinction between program evaluation and evaluation research. 
For many of the more descriptive areas addressed in the 
monograph, the distinction is not crucial. This includes areas 
that do not involve hypothesis testing (description of ageny 
mission, developing profiles of client characteristics and 
service needs, etc.). The issue is more pronouced when the 
evaluator attempts to test hypotheses concerning program 
effectiveness and, as a result, needs to control the research 
setting to a greater extents 

There has been continuing debate concerning the possibility 
of conducting outcome evaluation research at the program or 
agency level (see for example Ciarlo, 1982; Hargreaves, 1982; 
and Newman, 1982) . I'hct debate has centered on whether reliable 
and valid outcome d^ita can be obtained outside of the classic 
experimental model. 

Certainly there is an appropriate place for treatment 
research as defined by Hargreaves. This is targeted research 
aimed at identifying the impact of an intervej*tion that is 
delivered in relatively controlled settings. This means that 
the researcher has control over who receives the intervention 
and at least some of the potentially confounding influences that 
could distort the research findings. The most frequently used 
and most respected methodological design for treatment research 
is the classic experimental model with random assignnient to test 
and control group settings. 
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This is a slow process. It is also a costly one that 
requires considerable effort and expertise • However costly, 
treatment research plays a vital role in our understanding of 
treatment effectiveness. It is clear that we need more 
information on what types of interventions worksbest for what 
types of clients under what circumstances. 

Conducting this type of carefully controlled, long range 
research is virtually impossible at the program or agency 
level. Few agencies have the resources to support the staff 
required to conduct such studies and even fewer can set up the 
sort of sterile conditions that are necessary to avoid 
contaminating the research results. The fact remains, however, 
that program managers need to know something about the fate of 
their clients in order to manage their programs responsibly. 
They need to know whether their clients are spending too much 
time hospitalized, how many clients are finding employment, and 
whether clients are making any gains in terms of community 
functioning. The information has to be collected routinely and 
it has to be fed bacL to the people who make decisions about 
programs. This includes program directors and administrators, 
unit or service directors, program staff/ and legislators who 
control the program's funding. 

Obstacles to conducting sound outcome research remain. The 
available instruments are not perfect and our ability to isolate 
treatment interventions is far from complete. We agree with 
Ciarlo, however, when he writes (1982, p. 36) : 



....we need not wait until all such obstacles are removed 
and the measures in hand are perfect: they will never be 
perfect* A better strategy would be to choose carefully 
among available measurement techniques, recognize their 
specific limitations, compensate as well as possible with 
study designs or measure modifications, and begin the 
process of implementing outcome evaluation accountability in 
many of our programs. 

We have tried to select some instruments that will provide 
information to program managers and program planners < i, at the 
same time, will not impose an unreasonable reporting burden on 
direct service staff. 

Uses of Evaluation 

Program evaluation is a fairly futile activity unless the 
results of the research are disseminated widely and have an 
impact on programs. There are numerous techniques that 
evaluators can use to ensure the visibility of evaluation and 
monitoring results and the ultimate implementation of 
recommendations. The failure to give adequate consideration to 
a strategy for disseminating study results and for maximizing 
the chances that recommendations will be implemented is a 
serious shortcoming of many evaluation studies. In conducting 
service assessments, evaluators often focus too much energy on 
the study process itself. No matter how well the evaluation 
process is carried out, it is unlikely that the study will have 
its intended effects without a proportional expenditure of 
effort on dissemination. 

Perhaps the most direct way to disseminate evaluation 
results is through agency debriefing. This r ans simply that 
the evaluators or monitors communicate their findings and 
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recominendations directly to the program administrators and 
staff. This ensures that the results will recive a fair 
hearing. 

A second strategy is to include research recommendations in 
the agency's annual plan. This is a useful dissemination 
strategy since it establishes a benchmark against which the next 
year ^ s performance can be measured. 

Another strategy that may be useful on occasion is enlisting 
the support of state or local legislators. This is a useful 
technique that brings necessary information to the groups that 
control the allocation of resources. Legislatures are beginning 
to demand more accountability from programs that use public 
funds. Any information that can document the amount and types 
of services that have been bought with these dollars will 
certainly be welcome. Any information that documents service 
effectiveness will be even more welcome! 

Finally, evaluators and program administrators alike can 
make use of professional journals and association meetings to 
disseminate their findings to a wider range of persons with 
similar needs and concerns. In this way information on both 
successful programs and successful program evaluation strategies 
can be brought to the attention of the most relevant audience. 
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MEMBERSHIP SURVEY 



PART I: THIS SECTION IS TO BE ANSWERED BY ALL MEMBERS. 



How long have you been 


a member of Fellowship House? 








1985 




1984 


• 




N 


^ercent_ 


N ^Percent 


Less than one year 




25 


2^.8/. 


33 37.67. 


Longer than one year 




_5i_ 


70.2 






TOTAL 


"84 


100. OX 


10 r Too. 07. 


Sex 


• 












1 SOU 




1984 






M 
IN 


P *a K »~ 1^ ^ 




Male 




59 


57.8'/. 


65 63.77. 






_2a_ 


32.2 






TOTAL 


87 


100.07. 


"84 100707. 


Ethnicity 














1985 




1984 






M 


P ^ t"* 1^ »n ^ 


r\i Percent 


Anglo 






Dw ■ ^ /• 


61 £7.0% 


Black 




1 1 


IOC 


7 7.7 


Hispanic 




I 1 


1J« b 


12 13.? 


Other 












TOTAL 


s" 


100. o:: 


90 100.07. 


Age 














1985 




1984 






JL 


Percent^ 


_N_ _Ei!l££Qt. 


-25 




12 


14.57. 


N/A 


:5-34 




40 


48.2 




!5-44 




1& 


19.2 




5+ 






18.1 






TOTAL 


83 


100.07. 





5) The time it took me to become a member of Fellowship House 

1985 



was: 

1984* 



N 



Percent 



N 



Percent 



Too long 


15 


18. 1:: 


14 


15.47. 


Reasonable 


_68 


81.9 


-7Z_ 






TOTAL 83~ 


100.07. 


91 


100/0% 



t 1S84 had *'The time between my intake interview and acceptance into FH was:" 



143 



MEMBERSHIP SURVEY 
Page 2 



6) Do you feel that Fellowship House is open at times which are convenient' 



1985 



Yes 
No 

I am not sure what time 
Fellowship House is open, 



TOTAL 



Travel difficulties 
Cost of Social Activities 
Vocational Fees 
Residential Fees 



JL 


Percent 


JL 


70 


81.4-/. 


S4 


13 


15.1 


1 V 


3. 






86 


100. 0'/. 


9i 


\g services because 


of: 




1985 




_N_ 


_Percefit_ 




19 


21.6'/. 


22 


6 


6.8'/. 


8 


8 


9.17. 


5 


9 


10.2'/. 


10 



1984 



84.8*. 

10. \ \ 



100.0 



1984 



Percei t 



8) How long do you have to travel to reach Fellowship House? 



198J 



30 minutes or less 

31 ' 6C minutes 
1 ' -jur or more 



TOTAL 



_N_ 

70 
10 
_6. 
~86 



_Per|:ent_ 

81.47. 
11.6 

looToz 



2l.ojf 
7.6 
4.8 
9.5 



1984 

_N_ _P§L£iiltT 

67 69. 1 

14 14.4 

J.6 _lii.5 

37~ TooTq 



9) duplicate discarded: 
10) If you use FH vans, is service: 



1985 



1984 



Good 

Adequate 
Bad 



TOTAL 





Percent 




30 


42.37. 


50 


29 


40.8 


16 


_12_ 




6 


~71~ 


100.07. 


72 



_Perce!»t" 

69.4 
22 

8.3. 
100.0". 



11) Do you feel physically safe while you are at the Fellowship House Clubhouse 

1985 1584 



Al ways 

Sometimes 

Never 



TOTAL 



JL 


Percent 




-N- 


63 


75.07. 


yes 


85 


17 


20.2 






__4_ 




no 


8 


~84~ 


100.07. 




93 



_Perce2t. 

Too.o^ 
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12) Are you treated with courtesy and respect by FH receptionists and clerks? 

1985 1984 

'Yes, very much 57 S?.!"/. 68 70,17. ' 

Yes, somewhat 23 27.1 24 24.7 

Not very much 3 3.5 3 3.1 

Not at all 2 ^Zi.i_ ^JL- 

TOTAL 85" lOO.oj; 97 100.07. 



Do you feel comfortable talking with staff 


other than 


advisors 


or aides? 




1985 




1984 






Percent 


_N_ 


Percent 


Yes, very much 


43 


51.27. 


SI 


62.27. 


Yes, somewhat 


28 


33.3 


2S 


26.6 


No, not very much 


9 


10.7 


7 


7.1 


No, not at all 


__4_ 


4.8 


4_ 


__4i.l_ 




TOTAL 84" 


100.07. 


98~ 


100.07 



* Do you feel comfortable talking with Administrative Staff? 



Have you had problems talking to staff because: 








1985 






1984 




JL 


^ercent_ 


_N_ 


_ Percent 


They do not speak your 


5 


5.77 


9 


8. 67 


language? 










Other communication 


20 


22.77 


14 


13.37 


problems? 










Have you had any problems at 


Fellowship House because 


of your: 






1985 






1984 




N 


Percent , 


_N_ 


_P€rcent_ 


Race 


6 


6.87 


4 


3. 8X 


Culture 


5 


5.77 


8 


7.7 


Age 


7 


8.07 


10 


9.5 


Sex 


6 


6.87 


7 


6.7 


Education 


11 


12.57 






Income 


12 


13.67. 
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16) How helpful has Fellowship House staff been at aiding you in getting otiier 
community services (Food Stamps, Welfare, etc.)? 



1985 



Very helpful 
Slightly helpful 
Not helpful 
I did not need help 

TOTAL 

17) If a friend was in need of similar help, would you recommend him or her j 
for : I 



JL 


Percent 




46 


59.0'/. 


47 


8 


10.3 


16 


5 


6.4 


8 




_24i4_ 


„18_ 


78" 


loo.oi^ 


"89 



19S4 

Perceni 



52.8 



18.0 

9.0 ' 
.20i2j 

Too.o*-^ 



Social Program 



Yes 
No 



Vocational Program 



Yes 
No 



Residential Program 



1985 
N Percent 



1984 
N Percent 



TOTAL 



59 

_Z. 
66 



89.4 
100.07. 



80 
__4_ 
84" 



95. 2y! 
100.0*/ 



1985 



1984 





Percent 




_Percent_ 


55 


84.6V. 


66 


91.7/! 


_1Q_ 




__i 




65 


Too.ov. 


72 


looTo^ 



1985 



1984 







Jj_ 


Percent 


-Hi- 


Yes 




47 


82. 5V. 


ss 


No 






17.5 


10 




TOTAL 


57 


100. OV. 


65 



Percent 
84.67. 

Too.o'/J 



18) Overall, have the services you received helped you to deal more effectiv^l^ 
with your problems? 



198S 



1984 



Yes, a great deal 
Slightly helpful 
Not helpful 
I did not need help 



TOTAL 



_N_ 


Percent 


_N- 


Percent 


46 


55.4V. 


59 


61.5"/. 


27 


32.5 




34.4 


7 


8.5 


1 


1.0 






3 




83 


100. OV. 


96 


lOO.O'^ 



1 4' G 

4 
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19) During your metpbership at Fellowship House have services: 

1985 1984 



JL 


_Perce!it_ 


JL 


Percent 


43 


52.4*/. 


42 


43.37. 


30 


3&.& 


47 


48.5 




11.0 




8.2._ 


82 


100.07. 


97" 


100.0% 



Improved? 
Stayed the same 
Gotten worse 

TOTAL 

20) What changes have you noticed (good and bad) at Fellowship House since you 
entered program? 

Comments only 

?ART II. SOCIAL PROGRAM - THIS SECTION TO BE ANSWERED BY ALL MEMBERS. 

21) How would you rate the variety in Social Prograni activities? 

1985 1984 









_Percent_ 


_N _Perc.ent 


Quite a lot 




36 


45.67. 


32 35.57. 


Just right 






36.7 


39 43.3 


Not enough 






J.7-7_ 


-12- _2l4.1_ 




TOTAL 


~79 


100. OX 


90 100.0% 


Do you feel you have enough opportunity to help plan 


activities for Soci. 


Program? 














1985 




1984 






_N_ 


_PerceMt_ 




Yes, very much 




35 


• 45.67, 


37 40.77. 


Yes, somewhat 




27 


35.1 


39 42. 9 


No, not enough 




6 


7.8 


3 9.9 


No, not all all 




9 


11.7 


__6_ _li.6_ 






77 


100.07. 


85 100.0% 


Do you feel that FH provides 


enough 


activities during 


holiday seasons 


Christmas, Thanksgiving, Valentines Day, etc.)? 








1985 




1984 








_Percent_ 


Ji- _Perc_ent, 


Yes, very i... zh 




54 


' (:9.27. 


N/A 


Yes, somewhat 




16 


20.5 




No, not enough 




2 


2.6 




No, not all all 






— ZiZ- 






TOTAL 


78 


100. 0'Z 





Jf 47 

5 
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24) Are you treated with courtesy and respect by Social Program Stoff? 



1985 



Yes, very much 
Yes, most of the time 
No, not very much 
No, not all all 



TOTAL 





Percent 


_N_ 


52 


56.77. 


&3 


21 


26.9 


29 


4 


5.1 


2 


1 


1.3 


1 


78~ 


100, oii 


95 



1984 



66.3X 
30.5 
2.1 
1.1 
100.0/ 



25) Do you feel that Social Program staff are concerned when you have a 
personal problem? 



During Social Program hours: 



Very much 
Somewhat 
Not at all 

At Other Times: 



1985 



1984 



TOTAL 





Percent 


_N_ 


_Percent 


44 


57.9V. 


52 


55.9*/ 


26 


34.2 


37 


39.8 


_6 


7.9 


4 


4.3 


76~ 


100.07. 


93 


100.0?! 



1985 



1984 



Very much 
Somewhat 
Not at all 



Yes, very much 
Yes, somewhat 
No, not enough 
No, not at all 



27) What do you like best in Social Program? 
Comments only 

28) What do you like least in Social Program? 
Comments only 

29) Other Social Program Comments: 
Comments only 

i -1 b 

6 





H 


Percent 




Percent 




22 


50.0/. 


32 


50. 8Z 




15 


34.1 


26 


41.3 






_i5i9_ 


5 




TOTAL 


"44~ 


Too.o^ 


~63~ 


Too. ox 


Program has 


helped you socially? 








1985 




1934 




_N_ 


Percent 


_N_ 


_Percent 




32 


45.1 


46 


50.5'/ 




2a 


39.4 


32 


35.2 




5 


7.0 


5 


5.5 


TOTAL 


__6_ 
"71" 


100. OX 


91 


Too'dv. 
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PART III. RESIDENTIAL PROGRAM 

30) Which facility(s) have you been in? 



iSaS 1984 





JL 


Percent 


JL 


Percent 


Manor 


23 


26. IV. 


30 


28.67. 


Supervised 


38 


43. 2'A 


37 


35.27. 


Satellite 




22.77. 


16 


15.27. 



31) How much does living in Fellowship House Residence help you become more 
independent? 

iS85 1S84 





JL 


_Percent_ 


_N_ 


Percent, 


Very much 


28 


58.37. 


30 


63.87. 


Somewhat 


11 


22.9 


14 


29.8 


Very little 


4 


8.3 


2 


4.3 


Not at all 




10.4 




__2i.l_ 


TOTAL 


~48~ 


100.07. 


47 


Too.0% 


Are you treated with courtesy and respect by Residential 


Program staff? 




1985 




19S4 




N 


Percent 


_N_ 


Per cent . 


Yes, very much 


26 


49.17. 


25 


51.07. 


Yes, somewhat 


19 


35.8 


-:0 


40.9 


No, not very much 


3 


5.7 


3 


6.1 


No, not at all 


__5 


__9i4_ 


__1_ 


_2.0_ 


TOTAL 


53~ 


100.07. 


~49' 


ioo7o5: 


Is the number of Residential staff adequate? 










•■ 1985 




1984 






_N_ 


Percent 


_!^_ - 


Percent_ 


More than enough 


21 


42.97. 


12 


24.07. 


Adequate 


18 


36.7 


27 


54.0 


Hardly adequate 


6 


12.2 


7 


14.0 


Not enough 




8.2 


4 


__8i0_ 


TOTAL 


49 


100.07. 


fid 


10075% 


How do you rate the number of social 


activities in Residence? 






1985 




1984 






JL 


Percent 


N 


Percent 


More than enough 


13 


26.07. 


9 


19.67. 


Adequate 


21 


42.0 


23 


50.1 


Hardly adequate 


4 


8.0 


5 


10.9 


Not enough 


JZ- 


24.0 


9 


_12i6_ 


TOTAL 


50 


100.07. 


43~ 


Too7o*7 
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35) *Do you feel physically safe in Residence? 

1985 1984 

Percent 

Always 23 67.3-/. 29 60.4J 



Sometimes 





Percent 


-li- 


33 


S7.37. 


as 


15 


30.6 


18 


1_ 


_2.0 


,1,. 


43 


100.0% 


48 



37.5 



Never 1 Zi.0 L. 

VufAL 49 100.0% 48 100.0)! 

36) Are the furnishings in your apartment satisfactory? 

1385 1984 

. Percen 

Very Satisfactory 
Satisfactory 
Unsatisfactory 
Very Unsatisfactory 

TOTAL 

37) Are che rules for living in a Fellowship House residence reasonable? 

1S8S 1984 



Very reasonable 
Reasonable 
Unreasonable 
Very unreasonable 

'"QTAL 



_N_ 


^Percent 


_N_ 


20 


39.27. 


N/A 


19 


37.3 




11 


21.6 




1 


_2.0_ 




51 


100.0% 







Percent 


_N_ 


_Percent 


19 


39.£'/. 


15 


31.37. 


26 


54.2 


28 


58.3 


2 


4.2 


4 


8.2 


1_ 


__2U_ 


— 1_ 




4a~ 


100.0% 


48" 


100.07. 



Do you feel "at home" in your FH residence? 








1985 


19S4 


* 




JL. Percent 


JL 


Percent 


Yes 


32 65.37. 


28 


59.67. 


Somewhat 


10 20.4 


11 


23.4 


No 


7 14.3 


__a_ 


17^_ 


TOTAL 


49 100.07. 


47 


loo.oii 


Do you feel your apartment is home? 








Are there any other residential options besides the Manor, 


Supervised 


and Satellite that are needed. 










1985 


1984 






_N . ^Percent, 


_N_ 


Percent. 


No 


36 76.67. 


28 


65.17. 


Yes 


11 23.4 


15 


34.9 


TOTAL 




47 


100.07. 



bI 0 'J 
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40) What do you like best about Resiuence? 
Comraents only 

41) What do you like least about Residence? 
Comments only 

42) Other Res^'dential Cofr:nents: 
Comments only 

PART IV. VOCATIONAL PROGRAh 

43) Do you fe**l that you get the individual time you need from your 
advisor? 

1S85 1S84 







N 


Percent 


_N_ 


.Percent 


Always 




35 


. 61.47. 


42 


SO. 07. 


Sonetimes 




21 


36 • 8 


23 


32.9 


Never 




1_ 


_ii.a_ 


_ 5_ 






TOTAL 


57 


100. o5i 


~70 


100.0% 



44) How would you rate the service you have received in vocational program? 

1S85 1S84 

«Percent_ ^Percent^ 

Excellent 29 50.07. 37 53.67. 

Good IS 32.8 22 31.9 

Fair 7 ^2.1 7 10.2 

Poor 3 '5/2^ 3_ 4^3_ 

TOTAL 58~ 100. OX 69 10^.07. 

45) Are you treated with courtesy and respect by Vocational Program staff? 

1985 1984 

Yes, very much 
Yes, somewhat 
No, not very much 
N0| not at all 

TOTAL 

46) What do you like best about Vocational Program? 
Comments only 





_Percent_ 


_N_ 




39 


68.47. 


52 


75.47. 


15 


26.3 


IS 


21.8 


2 


3.5 


1 


1.4 


1 




1_ 


li4_ 


57 


100. o5: 


69 


Too. OX 



erIc 
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47) What do you like least about Vocational Program? 
Comments only 

IF YOU ATTEND WORK AREA ACTIVITIES THEN ANSWER QUESTIONS 4B-52. 

48) Is your wo^^k area activity is helping you to be productive? 

I 

1985 1984 
JL «P£!:cent_ JL -££!1££!1^ 

Yes, very much 27 48.2-/. 37 55. 2"/ 

Yes, somewhat 20 35.7 19 28,4 

No, not very much 5 5.7 6 9,0 

No, not at all „Zii« „Z-ii- 

TOTAL 56 100. OV. 67 100.07. 

49) Are you in the work area that you would like to be in? 

1985 1984 



Yes 
No 



a job? 



Yes, very much 
Yes, somewhat 
No, not very much 
No, not at all 



Yes, very much 
Yes, somewhat 
No, not very much 
No, not at all 



N 


Percent 


N 


Percent 


43 


79.6'/. 


N/A 






20.4 






TOTAL "54" 


100.07. 






area activity is 


helping you 


to prepare for 


1985 


1984 


_N_ 


Percent 


Jl_ - 


Percent 


23 


41.87. 


32 


44.47. 


19 


34.5 


21 


29.2 


9 


16.4 


12 


16.7 


__4_ 


_Zi3_ 


_z_ 




TOTAL 56' 


100.07. 


"67 


Too.ov: 


area activity is 


interesting 


and challenging? 


1985 


1984 


_N. 


_Percent_ 


_N_ 


Percent 


28 


51.9'/. 


So 


52.27. 


13 


24.1 


23 


33.3 


7 


13.0 


S 


7.2 


„6_ 




, ,5 




TOTAL 54 


100.07. 


69 


loo.oic 



15 2 
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52) Do you feel that Transitional Employment helps members to prepare for 
competitive employment? 



Yes, very much 
Yes, somewhat 
No, not very much 
No, not at all 



1985 1984 



TOTAL 





Percent 


N 


_ Percent^ 


31 


58.5'/. 


43 


67.2'/. 


15 


28.3 


IS 


23.4 


2 


3.8 


3 


4.7 


_5_ 




3 


»_4iZ- 


53 


100. OX 


£4 


100.0% 



IF YOU ATTEND TRANSITIONAL EMPLOYMENf (T 
53) bo you feel that there is enough vai 



Yes, very much 
Yes, somewhat 
No, not very much 
No, not at all 

TOTAL 

54) What other T.E. jobs would you like 
Comments only 

55) Does r.E. help you feel better about 



Yes, very much 
Yes, somewhat 
No, not very much 
No, not at all 

TOTAL 



E.), PLEASE ANSWER QUESTIONS 
iety in T.E. positions? 



1985 1984 





Percent 




Percent 


13 


37.1"/. 


9 


29.07. 


12 


34.3 


11 


35.5 


S 


17.1 


8 


25.8 


4 


11.4 


- 3_ 


9.7 


35~ 


100.07. 


31 


100.07. 



to see? 



yoursel f? 



1985 1984 





Percent 


_N_ 


_Percent 


13 


37. r/. 


17 


56.77. 


12 


34.3 


9 


30.0 


6 


17.1 


2 


6.8 


__4_ 


_ljL4_ 






35~ 


100. OX 


30 


iod~dv. 
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